SlideShare a Scribd company logo
1 of 37
Making Maintenance Manageable
  Using FME to Maintain FEMA’s National Flood Hazard Layer

Rob Gaines
Senior GIS Analyst, Zimmerman Associates (FEMA Ctr)




                                                             April 18th, 2012
What is the NFHL?


 FEMA’s National Flood Hazard Layer, part of the
  National Flood Insurance Program.

 Nationwide regulatory GIS dataset used to
  communicate flood risk information.
The Numbers


 Compiled from 2081 individual community or
  countywide flood study datasets (and growing).

 Average 95 county/communities updated per
  month.

 32 spatial layers, 18 lookup tables.

 Over 85 million records in total.
How is the NFHL Accessed?


 MapViewer: 100,000 – 200,000 requests per
  month.

 KMZ/WMS: 3 – 6 million requests per month.

 NFHL Extract (via state shapefiles).
What is the NFHL?
What is the NFHL?
What is the NFHL?
What is the NFHL?
FIRM/NFHL Production Structure




FIRM Database        PTS rFHL          CDS NFHL
 County Dataset   Regional Dataset   National Dataset
FIRM/NFHL Production Structure
Data Going Effective 4/3/2012
Challenges


 Data produced at community level.

 Initial QA/QC process not intended for NFHL.

 Bureaucracy: 10 exceptions to every rule.

 Far removed from data producers.

 Inherited systems.
What Is FME Used For?


 QA/QC of data submissions.

 Data replication.

 Extraction.

 Report generation / tracking.

 LOMA Point creation.
Basic QA/QC Checks


 All records have DFIRM_ID value.
 All required layers are present.
 No duplicate geometry.
 Flood hazard lines & areas have valid flood zone
  values.
 Effective date information is correct.
 New data contains all currently effective LOMRs.
QA/QC FME Model
QA/QC: DFIRM_ID Value


 Uses FeatureMerger transformer to check that all
  records have DFIRM_ID value and all DFIRM_ID
  values match a record in the STUDY_INFO table.
QA/QC: Required Layers


 Uses ListPopulator and ListExploder along with
  FeatureMerger to create a list of all layers present
  in submission and then check them against the
  list of required layers.
QA/QC: Duplicate Geometry


 Concatenates a number of geometric attributes
  (such as coordinate count, length / area, etc)
  with flat attributes to detect duplicate features
  and filter them out.
QA/QC: Valid Values


 Uses the AttributeFilter to detect any invalid flood
  zone values.
QA/QC: LOMR Checker


 Compares submission LOMR layer with current
  NFHL LOMR layer to detect any missing
  information.
Sample QA/QC Sheet
Data Staging & Replication

            Staging
 NFHL
 Data
              DB


 NFHL         FME
 Data        Loader                NFHL SDE


 NFHL
 Data         FME       FME
             Logger   Replicator




             Log      MS Access
                      Interface
             DB
StagingLoad Model
StagingLoad Model: Logger
Replicator Access Query
Replicator Access Form
Extraction


 Extract all NFHL records into state-based
  shapefiles.

 Generate reports such as panel and LOMR count
  for each state.

 QC: Check current extract against prior extract to
  ensure completeness of layers.
Extraction: QC FME Model


 Runs through each state extract.

 Uses ListBuilder to build a list of each state’s
  layers and check them against previous month’s.

 Generates Excel report.
Extraction: QC Sheet
Reporting & Tracking


 Inventory of current NFHL holdings.

 Status of currently effective LOMRs.

 QC of existing NFHL data.
   Missing layers.
   Invalid values.
   Obsolete data.
NFHL Inventory Report
NFHL LOMR Status Report
LOMA Mapper


 Daily process which converts flat data with lat/lon
  attributes to geospatial data in SDE.

 Replaced custom-developed VB application.

 Original application: 18+ hour runtime.
  FME Model: <10 minute runtime.
LOMA Mapper Flowchart
LOMA Mapper FME Model
Into the Future… FME Server


 Internal
   Automated data submission & QA/QC process.

 External
   Clip/Zip/Ship via MapViewer.
   Custom extracts.
   Custom on-demand GIS services.
Thank You!


 Questions?

 For more information:
   Rob Gaines: robert.gaines@riskmapcds.com
    Zimmerman Associates Inc. (FEMA Contractor)

   FEMA Map Service Center: http://msc.fema.gov

More Related Content

Viewers also liked

Lightning Talk: FME 2012 Tips and Tricks
Lightning Talk: FME 2012 Tips and TricksLightning Talk: FME 2012 Tips and Tricks
Lightning Talk: FME 2012 Tips and TricksSafe Software
 
Using FME for the City of Palo Alto Topobase Implentation
Using FME for the City of Palo Alto Topobase ImplentationUsing FME for the City of Palo Alto Topobase Implentation
Using FME for the City of Palo Alto Topobase ImplentationSafe Software
 
What's Great in FME 2012
What's Great in FME 2012What's Great in FME 2012
What's Great in FME 2012Safe Software
 
FME Lightning Talk: Raster
FME Lightning Talk: RasterFME Lightning Talk: Raster
FME Lightning Talk: RasterSafe Software
 
FME Lightning Talk: LiDAR
FME Lightning Talk: LiDARFME Lightning Talk: LiDAR
FME Lightning Talk: LiDARSafe Software
 
Hydraulic Modelling with GIS Data
Hydraulic Modelling with GIS DataHydraulic Modelling with GIS Data
Hydraulic Modelling with GIS DataSafe Software
 
Managing Spatial Data for Telecommunications Using FME
Managing Spatial Data for Telecommunications Using FMEManaging Spatial Data for Telecommunications Using FME
Managing Spatial Data for Telecommunications Using FMESafe Software
 
FME Lightning Talk: Esri Geodatabase Templates
FME Lightning Talk: Esri Geodatabase TemplatesFME Lightning Talk: Esri Geodatabase Templates
FME Lightning Talk: Esri Geodatabase TemplatesSafe Software
 
GIS Data Management for Large Infrastructure Projects
GIS Data Management for Large Infrastructure ProjectsGIS Data Management for Large Infrastructure Projects
GIS Data Management for Large Infrastructure ProjectsSafe Software
 
FME Lightning Talk: FMEpedia
FME Lightning Talk: FMEpediaFME Lightning Talk: FMEpedia
FME Lightning Talk: FMEpediaSafe Software
 
Improving Productivity Using FME Workbench
Improving Productivity Using FME WorkbenchImproving Productivity Using FME Workbench
Improving Productivity Using FME WorkbenchSafe Software
 
FME Lightning Talk: Multiprocessing
FME Lightning Talk: MultiprocessingFME Lightning Talk: Multiprocessing
FME Lightning Talk: MultiprocessingSafe Software
 

Viewers also liked (14)

FME Store
FME StoreFME Store
FME Store
 
Lightning Talk: FME 2012 Tips and Tricks
Lightning Talk: FME 2012 Tips and TricksLightning Talk: FME 2012 Tips and Tricks
Lightning Talk: FME 2012 Tips and Tricks
 
Using FME for the City of Palo Alto Topobase Implentation
Using FME for the City of Palo Alto Topobase ImplentationUsing FME for the City of Palo Alto Topobase Implentation
Using FME for the City of Palo Alto Topobase Implentation
 
What's Great in FME 2012
What's Great in FME 2012What's Great in FME 2012
What's Great in FME 2012
 
FME & Governement
FME & GovernementFME & Governement
FME & Governement
 
FME Lightning Talk: Raster
FME Lightning Talk: RasterFME Lightning Talk: Raster
FME Lightning Talk: Raster
 
FME Lightning Talk: LiDAR
FME Lightning Talk: LiDARFME Lightning Talk: LiDAR
FME Lightning Talk: LiDAR
 
Hydraulic Modelling with GIS Data
Hydraulic Modelling with GIS DataHydraulic Modelling with GIS Data
Hydraulic Modelling with GIS Data
 
Managing Spatial Data for Telecommunications Using FME
Managing Spatial Data for Telecommunications Using FMEManaging Spatial Data for Telecommunications Using FME
Managing Spatial Data for Telecommunications Using FME
 
FME Lightning Talk: Esri Geodatabase Templates
FME Lightning Talk: Esri Geodatabase TemplatesFME Lightning Talk: Esri Geodatabase Templates
FME Lightning Talk: Esri Geodatabase Templates
 
GIS Data Management for Large Infrastructure Projects
GIS Data Management for Large Infrastructure ProjectsGIS Data Management for Large Infrastructure Projects
GIS Data Management for Large Infrastructure Projects
 
FME Lightning Talk: FMEpedia
FME Lightning Talk: FMEpediaFME Lightning Talk: FMEpedia
FME Lightning Talk: FMEpedia
 
Improving Productivity Using FME Workbench
Improving Productivity Using FME WorkbenchImproving Productivity Using FME Workbench
Improving Productivity Using FME Workbench
 
FME Lightning Talk: Multiprocessing
FME Lightning Talk: MultiprocessingFME Lightning Talk: Multiprocessing
FME Lightning Talk: Multiprocessing
 

Similar to Making Maintenance Manageable

SoftQL - Telecom Triage Services
SoftQL - Telecom Triage Services SoftQL - Telecom Triage Services
SoftQL - Telecom Triage Services Amar Uppalapati
 
Venturing into Hadoop Large Clusters
Venturing into Hadoop Large ClustersVenturing into Hadoop Large Clusters
Venturing into Hadoop Large ClustersVARUN SAXENA
 
Venturing into Large Hadoop Clusters
Venturing into Large Hadoop ClustersVenturing into Large Hadoop Clusters
Venturing into Large Hadoop ClustersNaganarasimha Garla
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureVARUN SAXENA
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureVARUN SAXENA
 
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords   The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords Stephan Ewen
 
Centralized logging with Flume
Centralized logging with FlumeCentralized logging with Flume
Centralized logging with FlumeRatnakar Pawar
 
Mainframe Virtual User Group Summer 2013
Mainframe Virtual User Group Summer 2013Mainframe Virtual User Group Summer 2013
Mainframe Virtual User Group Summer 2013Serena Software
 
FME User Conference Presentation
FME User Conference PresentationFME User Conference Presentation
FME User Conference Presentationcwise
 
Fmeuc Presentation Hrsd3
Fmeuc Presentation Hrsd3Fmeuc Presentation Hrsd3
Fmeuc Presentation Hrsd3guest5c50492
 
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...Yahoo Developer Network
 
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...Flink Forward
 
Environment Canada's Data Management Service
Environment Canada's Data Management ServiceEnvironment Canada's Data Management Service
Environment Canada's Data Management ServiceSafe Software
 
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017Till Rohrmann
 
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...Ververica
 
Training Slides: 153 - Working with the CLI
Training Slides: 153 - Working with the CLITraining Slides: 153 - Working with the CLI
Training Slides: 153 - Working with the CLIContinuent
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large ScaleVerverica
 

Similar to Making Maintenance Manageable (20)

FME to the Rescue
FME to the RescueFME to the Rescue
FME to the Rescue
 
SoftQL - Telecom Triage Services
SoftQL - Telecom Triage Services SoftQL - Telecom Triage Services
SoftQL - Telecom Triage Services
 
Venturing into Hadoop Large Clusters
Venturing into Hadoop Large ClustersVenturing into Hadoop Large Clusters
Venturing into Hadoop Large Clusters
 
Venturing into Large Hadoop Clusters
Venturing into Large Hadoop ClustersVenturing into Large Hadoop Clusters
Venturing into Large Hadoop Clusters
 
Lee.stat
Lee.statLee.stat
Lee.stat
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and Future
 
Application Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and FutureApplication Timeline Server - Past, Present and Future
Application Timeline Server - Past, Present and Future
 
The Stream Processor as a Database Apache Flink
The Stream Processor as a Database Apache FlinkThe Stream Processor as a Database Apache Flink
The Stream Processor as a Database Apache Flink
 
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords   The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
 
Centralized logging with Flume
Centralized logging with FlumeCentralized logging with Flume
Centralized logging with Flume
 
Mainframe Virtual User Group Summer 2013
Mainframe Virtual User Group Summer 2013Mainframe Virtual User Group Summer 2013
Mainframe Virtual User Group Summer 2013
 
FME User Conference Presentation
FME User Conference PresentationFME User Conference Presentation
FME User Conference Presentation
 
Fmeuc Presentation Hrsd3
Fmeuc Presentation Hrsd3Fmeuc Presentation Hrsd3
Fmeuc Presentation Hrsd3
 
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by U...
 
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
Flink Forward SF 2017: Joe Olson - Using Flink and Queryable State to Buffer ...
 
Environment Canada's Data Management Service
Environment Canada's Data Management ServiceEnvironment Canada's Data Management Service
Environment Canada's Data Management Service
 
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
 
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
 
Training Slides: 153 - Working with the CLI
Training Slides: 153 - Working with the CLITraining Slides: 153 - Working with the CLI
Training Slides: 153 - Working with the CLI
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large Scale
 

More from Safe Software

From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemSafe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISSafe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersSafe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsSafe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategySafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Safe Software
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMESafe Software
 

More from Safe Software (20)

From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FME
 

Recently uploaded

SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700
 
Google AI Hackathon: LLM based Evaluator for RAG
Google AI Hackathon: LLM based Evaluator for RAGGoogle AI Hackathon: LLM based Evaluator for RAG
Google AI Hackathon: LLM based Evaluator for RAGSujit Pal
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024Results
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 

Recently uploaded (20)

SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
 
Google AI Hackathon: LLM based Evaluator for RAG
Google AI Hackathon: LLM based Evaluator for RAGGoogle AI Hackathon: LLM based Evaluator for RAG
Google AI Hackathon: LLM based Evaluator for RAG
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 

Making Maintenance Manageable

  • 1. Making Maintenance Manageable Using FME to Maintain FEMA’s National Flood Hazard Layer Rob Gaines Senior GIS Analyst, Zimmerman Associates (FEMA Ctr) April 18th, 2012
  • 2. What is the NFHL?  FEMA’s National Flood Hazard Layer, part of the National Flood Insurance Program.  Nationwide regulatory GIS dataset used to communicate flood risk information.
  • 3. The Numbers  Compiled from 2081 individual community or countywide flood study datasets (and growing).  Average 95 county/communities updated per month.  32 spatial layers, 18 lookup tables.  Over 85 million records in total.
  • 4. How is the NFHL Accessed?  MapViewer: 100,000 – 200,000 requests per month.  KMZ/WMS: 3 – 6 million requests per month.  NFHL Extract (via state shapefiles).
  • 5. What is the NFHL?
  • 6. What is the NFHL?
  • 7. What is the NFHL?
  • 8. What is the NFHL?
  • 9. FIRM/NFHL Production Structure FIRM Database PTS rFHL CDS NFHL County Dataset Regional Dataset National Dataset
  • 12. Challenges  Data produced at community level.  Initial QA/QC process not intended for NFHL.  Bureaucracy: 10 exceptions to every rule.  Far removed from data producers.  Inherited systems.
  • 13. What Is FME Used For?  QA/QC of data submissions.  Data replication.  Extraction.  Report generation / tracking.  LOMA Point creation.
  • 14. Basic QA/QC Checks  All records have DFIRM_ID value.  All required layers are present.  No duplicate geometry.  Flood hazard lines & areas have valid flood zone values.  Effective date information is correct.  New data contains all currently effective LOMRs.
  • 16. QA/QC: DFIRM_ID Value  Uses FeatureMerger transformer to check that all records have DFIRM_ID value and all DFIRM_ID values match a record in the STUDY_INFO table.
  • 17. QA/QC: Required Layers  Uses ListPopulator and ListExploder along with FeatureMerger to create a list of all layers present in submission and then check them against the list of required layers.
  • 18. QA/QC: Duplicate Geometry  Concatenates a number of geometric attributes (such as coordinate count, length / area, etc) with flat attributes to detect duplicate features and filter them out.
  • 19. QA/QC: Valid Values  Uses the AttributeFilter to detect any invalid flood zone values.
  • 20. QA/QC: LOMR Checker  Compares submission LOMR layer with current NFHL LOMR layer to detect any missing information.
  • 22. Data Staging & Replication Staging NFHL Data DB NFHL FME Data Loader NFHL SDE NFHL Data FME FME Logger Replicator Log MS Access Interface DB
  • 27. Extraction  Extract all NFHL records into state-based shapefiles.  Generate reports such as panel and LOMR count for each state.  QC: Check current extract against prior extract to ensure completeness of layers.
  • 28. Extraction: QC FME Model  Runs through each state extract.  Uses ListBuilder to build a list of each state’s layers and check them against previous month’s.  Generates Excel report.
  • 30. Reporting & Tracking  Inventory of current NFHL holdings.  Status of currently effective LOMRs.  QC of existing NFHL data.  Missing layers.  Invalid values.  Obsolete data.
  • 33. LOMA Mapper  Daily process which converts flat data with lat/lon attributes to geospatial data in SDE.  Replaced custom-developed VB application.  Original application: 18+ hour runtime. FME Model: <10 minute runtime.
  • 36. Into the Future… FME Server  Internal  Automated data submission & QA/QC process.  External  Clip/Zip/Ship via MapViewer.  Custom extracts.  Custom on-demand GIS services.
  • 37. Thank You!  Questions?  For more information:  Rob Gaines: robert.gaines@riskmapcds.com Zimmerman Associates Inc. (FEMA Contractor)  FEMA Map Service Center: http://msc.fema.gov