SlideShare une entreprise Scribd logo
1  sur  17
CONNECT. TRANSFORM. AUTOMATE.
Improving Data Compliancy
Using FME
David van Riel
GIS Technologist, City of Kitchener
City of Kitchener Information
 1 hour west of Toronto.
 200,000+ people.
 Centralized GIS in ESRISDE environment.
 Maintain ~400 GIS layers, many linked and/or
contribute to other systems.
 User of FME for last ~8 years. Initially for data
exporting and transforming, but increasingly
more for data loading and manipulation.
Need for Complete and Correct Data
 Asset management:
 Public Sector Accounting Board (PSAB) 3150 for
reporting of tangible capital assets.
 SAP for city financial and accounting needs.
 For work order management, inspection
programs, and reporting being done through, or
housed in, Cityworks.
 As usual any and all other
mapping, analysis, reporting, and exporting done
on city’s GIS asset information.
Infrastructure Features:
Former Data Capture Process
Majority of:
 Sanitary sewer
 Storm water sewer
 Water
 Geographic capture
= retrace features
Infrastructure Features:
Former Data Capture Process
Is information present?
Infrastructure Features:
Former Data Capture Process
Where
is the
information?
Infrastructure Features:
Former Data Capture Process
 Slow.
 Prone to errors.
 Data entered multiple times.
 Unhappy customers.
Digital Submission Capability/Ability
 CAD characteristics (CAD Standards Checker)
 Complete attribute data (Attribute Checker)
 Correct attribute data (Attribute Checker)
 Correct geographic placement (Topology Checker)
*
*
* Relative, not absolute
Digital Submission Compliancy Checkers
 CAD Standards, Attribute, and Topology Checkers
developed by Consortech.
 FME reads consultant submitted AutoCAD Map 3D
files (with object data) and Excel files containing
basic checker settings.
 Requirements for all checkers:
1. GIS or AutoCAD, and text or Excel outputs.
2. Easily customizable settings for changing needs.
CAD Standards Checker
Attribute Checker
Topology Checker
12 rules:
 node edge snap
 node end snap
 floating node
 duplicate node
 line end snap
 line edge snap
 crossed line
 line end node
 duplicate line
 floating service
 one line two nodes
 both line ends snap
Loading Data into CoK Database
 After drawing has passed all checkers the data is
ready to be exported from AutoCAD and loaded
into the GIS database.
 Use AutoCAD epf (export profile) files to export
to shapefiles, then using ArcGIS Model Builder to
load data into database.
Manual Checks and Procedures
 Check connectivity of new features to existing
features.
 Register .pdf files in EDRA (in house Electronic
Document Registration Application) and create
linked boundaries in GIS.
 Fill in fields not provided by consultants
(SOURCE, SOURCE_DATE, ROADSEGMENTID, QUARTER_GRID_ID, UP_MA
NHOLEID, etc…).
Digital Submission / Data
Compliancy Benefits
 Time saving (solves slow method).
 Accuracy/completeness of data (solves prone to
error issue).
 Defined deliverables with respect all parties
(solves multiple data entry issue).
 Everyone is happy!
What’s Next?
 Synchronize AutoCAD template to GIS database
(constantly changing).
 More/improved database loading automation and
automate manual GIS fields where possible
(using FME for complete process?).
 Incorporate Region of Waterloo projects.
 Apply checkers to other processes (gas utility –
AutoCAD training late fall 2013 and first digital
submissions spring 2014).
 FME Server.
Thank You!
 Questions?
 For more information: David.vanRiel@Kitchener.ca
City of Kitchener
 AutoCAD template at
http://www.kitchener.ca/en/businessinkitchener/Development_manual.a
sp (or search for “development manual”)
 http://www.kitchener.ca/en/
 https://www.facebook.com/cityofkitchener
 https://twitter.com/kitchenernews
 http://www.flickr.com/photos/cityofkitchener
 http://www.youtube.com/user/cityofkitchener

Contenu connexe

Plus de Safe Software

Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISSafe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersSafe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsSafe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategySafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Safe Software
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMESafe Software
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Safe Software
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Safe Software
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleSafe Software
 
Democratizing Transformer Creation in FME
Democratizing Transformer Creation in FMEDemocratizing Transformer Creation in FME
Democratizing Transformer Creation in FMESafe Software
 
Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Safe Software
 
Datendrehscheibe im eGovernment
Datendrehscheibe im eGovernmentDatendrehscheibe im eGovernment
Datendrehscheibe im eGovernmentSafe Software
 

Plus de Safe Software (20)

Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FME
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) People
 
Democratizing Transformer Creation in FME
Democratizing Transformer Creation in FMEDemocratizing Transformer Creation in FME
Democratizing Transformer Creation in FME
 
Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...
 
Datendrehscheibe im eGovernment
Datendrehscheibe im eGovernmentDatendrehscheibe im eGovernment
Datendrehscheibe im eGovernment
 

Dernier

Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessPixlogix Infotech
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 

Dernier (20)

Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 

Improving Data Compliancy Using FME

  • 1. CONNECT. TRANSFORM. AUTOMATE. Improving Data Compliancy Using FME David van Riel GIS Technologist, City of Kitchener
  • 2. City of Kitchener Information  1 hour west of Toronto.  200,000+ people.  Centralized GIS in ESRISDE environment.  Maintain ~400 GIS layers, many linked and/or contribute to other systems.  User of FME for last ~8 years. Initially for data exporting and transforming, but increasingly more for data loading and manipulation.
  • 3. Need for Complete and Correct Data  Asset management:  Public Sector Accounting Board (PSAB) 3150 for reporting of tangible capital assets.  SAP for city financial and accounting needs.  For work order management, inspection programs, and reporting being done through, or housed in, Cityworks.  As usual any and all other mapping, analysis, reporting, and exporting done on city’s GIS asset information.
  • 4. Infrastructure Features: Former Data Capture Process Majority of:  Sanitary sewer  Storm water sewer  Water  Geographic capture = retrace features
  • 5. Infrastructure Features: Former Data Capture Process Is information present?
  • 6. Infrastructure Features: Former Data Capture Process Where is the information?
  • 7. Infrastructure Features: Former Data Capture Process  Slow.  Prone to errors.  Data entered multiple times.  Unhappy customers.
  • 8. Digital Submission Capability/Ability  CAD characteristics (CAD Standards Checker)  Complete attribute data (Attribute Checker)  Correct attribute data (Attribute Checker)  Correct geographic placement (Topology Checker) * * * Relative, not absolute
  • 9. Digital Submission Compliancy Checkers  CAD Standards, Attribute, and Topology Checkers developed by Consortech.  FME reads consultant submitted AutoCAD Map 3D files (with object data) and Excel files containing basic checker settings.  Requirements for all checkers: 1. GIS or AutoCAD, and text or Excel outputs. 2. Easily customizable settings for changing needs.
  • 12. Topology Checker 12 rules:  node edge snap  node end snap  floating node  duplicate node  line end snap  line edge snap  crossed line  line end node  duplicate line  floating service  one line two nodes  both line ends snap
  • 13. Loading Data into CoK Database  After drawing has passed all checkers the data is ready to be exported from AutoCAD and loaded into the GIS database.  Use AutoCAD epf (export profile) files to export to shapefiles, then using ArcGIS Model Builder to load data into database.
  • 14. Manual Checks and Procedures  Check connectivity of new features to existing features.  Register .pdf files in EDRA (in house Electronic Document Registration Application) and create linked boundaries in GIS.  Fill in fields not provided by consultants (SOURCE, SOURCE_DATE, ROADSEGMENTID, QUARTER_GRID_ID, UP_MA NHOLEID, etc…).
  • 15. Digital Submission / Data Compliancy Benefits  Time saving (solves slow method).  Accuracy/completeness of data (solves prone to error issue).  Defined deliverables with respect all parties (solves multiple data entry issue).  Everyone is happy!
  • 16. What’s Next?  Synchronize AutoCAD template to GIS database (constantly changing).  More/improved database loading automation and automate manual GIS fields where possible (using FME for complete process?).  Incorporate Region of Waterloo projects.  Apply checkers to other processes (gas utility – AutoCAD training late fall 2013 and first digital submissions spring 2014).  FME Server.
  • 17. Thank You!  Questions?  For more information: David.vanRiel@Kitchener.ca City of Kitchener  AutoCAD template at http://www.kitchener.ca/en/businessinkitchener/Development_manual.a sp (or search for “development manual”)  http://www.kitchener.ca/en/  https://www.facebook.com/cityofkitchener  https://twitter.com/kitchenernews  http://www.flickr.com/photos/cityofkitchener  http://www.youtube.com/user/cityofkitchener

Notes de l'éditeur

  1. Just to give you some information on our organization. We are a city of 200,000 west of Toronto.We have a centralized GIS in an ESRISDE environment and in this environment we have many layers that are used, use, or are linked to information from many systems. AMANDA, Cityworks, SAP, Stormwater Rate, and CIS to name a few.We have been using FME for a while and each year we increase our use of this tool and the complexity of the work that it does.
  2. Over the last few years our organization has experienced an increase in the interest and scrutiny of GIS data. While there has always been an effort to have correct and complete data there are always layers, geographic areas, or attribute fields that have slipped through the cracks for various reasons.There are many reasons why the GIS data at our city needs to be complete and correct as you can see here. But the reason in general terms is that there is a continual increase of GIS data being integrated into everyone’s everyday work. More than ever non-GIS data needs to be linked to GIS features and shown together, reports are being generated from GIS data, and decisions are being made based on these reports.
  3. The types of features most viewed, most reported, and most decisions made based on those views and reports has been engineering infrastructure features – sanitary sewer, storm water sewer and water utilities. All these features are submitted together in drawings by consultants, in the case of new subdivisions, or by internal city staff, for reconstruction projects.To give you a little background on how layers, geographic areas, or attribute fields could have slipped through the cracks before the implementation of the digital submission process you’d need to understand how the data was previously captured. Scanned tifs or digitally produced pdf files were supplied to GIS staff. It was expected that most of the information needed in the GIS layers would be provided in the drawings.NEXT POINTThe first thing was to capture the location of the features and that meant georeferencing the plan and doing some heads up digitizing. Rarely was the AutoCAD file provided to GIS, it was never georeferenced, and most features would need major editing to be usable anyway. For example pipes would be snapped to the edge of manhole blocks, and not the center of the block where the point would be captured in GIS.
  4. Next would be to capture the attribute information from those drawings.We can see from the earlier drawing example screenshot on the left that not all information was provided in the past. The diameter and slope are all that is supplied in terms of attribute data for this example.However, the screenshot on the right shows a more recent drawing, a digitally produced pdf file, that provides more information. One question is: are all the pieces of information that are needed there? They should be…
  5. …and most of the time they were. If all the pieces are provided in the drawings, the next question was: where are they?This is a screenshot of one of the pages of documentation I was handed when I first started with the city back in 2005. It shows where to typically get the needed information for a storm water pipe off of a plan and profile drawing. Not all plan and profile drawings were the same though. The location of the information could be anywhere. It could depend on the submitting consultant; the amount of area being covered in a drawing or project; or if there was room for the information in the main part of the drawing or if the information was put in the surround notes. Sometimes information would have to be gotten off of different drawings. For example a pipe invert might be only shown on a manhole detail drawing and not on the plan and profile drawing.Basically look at all the drawings provided and find the information you need.
  6. So the difficulties associated with the old method of capturing the infrastructure data were a hindrance for the smooth incorporation of GIS data in the viewing, reporting, decision making, and daily work flow processes for others in the city.NEXT POINTThe old method was slow. City staff were needing a faster turn around time. They needed that data. A typical sized subdivision would take a week of staff time to capture in GIS.NEXT POINTThe old method was prone to errors. Either the correct drawings with the information were not provided, the information was not provided in any drawings, or GIS staff couldn’t find or interpret it.NEXT POINTThe old method wasted time. Why was this information being input again in a different format? Shouldn’t this informationbe entered once by the ones making the drawings? The job of GIS staff should be to facilitate the importing of the data, but not the searching for or interpretation of that data.NEXT POINTNo one was happy. The data in GIS wasn’t as reliable as it should have been and no one used it if they didn’t have to. Engineering staff had their own repository of drawings and data. Operations staff relied on hard copy inspection and maintenance forms for data that was missing in GIS, or for city wide reports. Consultants needed the GIS data for new construction planning or for any consulting work and they couldn’t get it. And worst of all, from GIS staff’s point of view, it felt like the GIS section was getting the majority of the blame for the missing data.
  7. In the GIS section at the City of Kitchener the idea had always been discussed that wouldn’t it be nice to be able to automatically check and load the data into the database. Certainly there must be technologies to enable this? We knew there were, but never have the right conditions come along: namely strong non-GIS staff support and need for the benefits of digital submission, and a tool like FME to allow for data compliancy checks.NEXT POINTCAD standards could be checked. For many years Consultants were required to use an AutoCAD template file for these standards and they had been following them properly. But the ability to check the colour, line type, and line weight of AutoCAD entities on a layer – the properties that allow for correct looking digital plans to be produced – would ensure proper looking pdf files.NEXT POINTAttribute data could be inspected and verified. This would be something entirely new for us. Are all the necessary attribute fields provided, and are the values that are provided in those fields valid.NEXT POINTFinally, the placement of the submitted AutoCAD entities can be checked with respect to other submitted features.NEXT POINTIt must be noted that the attribute and placement correctness are what I call relative. This means that it is easy to check if the attribute field or placement of an AutoCAD entity follows a rule, but not if the value or placement is necessarily correct.
  8. As I said, the right conditions came into place to allow for digital submission implementation and therefore data compliancy checks. However, the city lacked the staff resources to make this a reality.NEXT POINTIt was around this time that we saw a presentation by Consortech concerning the use of FME for checking CAD standards. We approached this consultant company and asked if it was possible to create the kinds of data compliancy checks we were thinking of. They assured us it was, and out of this grew what we refer to as the CAD Standards, Attribute, and Topology Checkers.NEXT POINTThese checkers use FME scripts to read consultant submitted AutoCAD Map 3D and checker settings files to produce error outputs that can be used by these consultants to rectify any issues that are reported.NEXT POINTThere were 2 major requirements the city askedfor in all the checkers.One was that they had to produce some form of GIS or CAD output for visual purposes, and a text report. Either of these should enable the submitter of the data to easily find the issue AutoCAD entities.NEXT POINTThe other requirement was that because of the constant changes concerning the features being captured and the attributes associated with them, the basic settings for the checkers had to be customizable.
  9. Here are some screenshots for the CAD Standards Checker. Consortech had already developed this tool for similar purposes for other clients.On the left is an example of the AutoCAD information for an AutoCAD entity on the water main layer: it’s colour and line type.On the right is the CAD Standards Checker Manager Excel file that contains what the properties for AutoCAD entities on specific AutoCAD layers should be. Colour, line type and line weight properties can be checked.NEXT POINTThe FME CAD Standards Checker is run and uses these two inputs. Any issues are reported out to a summary and a detailed error text file.NEXT POINTThe checker also produces a CAD file that contains AutoCAD geometric shape entities grouped by a combination of input feature entity type and error issue at the center point of each issue entity. The properties for these geometric shape entities are set in the manager file. This CAD error file is georeferenced and can be imported into the original file.NEXT POINTIf no entities are returned as issues then the correct looking pdf files can be produced. These are important to our engineering and operations staff as not all information is captured in GIS and the drawings are often used to provide a quick overall view of the infrastructure in an area.
  10. Here are some screenshots for the Attribute Checker.In AutoCAD Map 3D there is the ability to attach object data, which are basic attribute tables, to AutoCAD entities. This screenshot shows the structure of the WAT_MAIN object data table on the left and it attached to an entity on the water main layer on the right with the attributes filled in.NEXT POINTThe Attribute Checker uses a csv file produced in Excel that identifies what object data field names should be attached to the entities on specific AutoCAD layers. Additional checks on each field can be performed – minimum and maximum character lengths, field data types, minimum and maximum number values, pick list restrictions, and whether the field is required to have a value or not.NEXT POINTThe FME Attribute Checker is run and uses these two inputs and, as with the CAD Standards Checker, any issues found by the Attribute Checker are reported out to a summary and a detailed error text file.NEXT POINTA shapefile is also produced that shows the location of the issue entities. This shapefile contains the same information that is present in the detailed error file and can be brought into AutoCAD Map 3D if needed.
  11. And finally the Topology Checker.There are 12 topology rules that the Topology Checker tests for and the layers that each rule tests on can be customized in an Excel file. Using this topology manager file the FME Topology Checker tests lines and points in the submitted file.NEXT POINTA detailed error Excel file and a shapefile are produced. Again, this shapefile contains the same information that is present in the detailed error file and can be brought into AutoCAD Map 3D if needed.
  12. If the submitted file passes all checkers without issues the drawing is approved and the data can then be loaded into the GIS database.NEXT POINTCurrently a combination of AutoCAD export profile files and ArcGIS’s Model Builder are used to extract and load the data. These are run manually but we are looking into ways to increase the automation of this step.
  13. After the data is loaded into the database there are always final edits and checks to perform.NEXT POINTChecking for logical connections with existing features being the most important of them. It is always assumed that the new features being submitted are in the proper location and the existing features are snapped to them.NEXT POINTThe digitally produced pdf drawing files are registered into an in-house document management system and extent boundaries are created in GIS for each of the drawings. These boundaries are then linked to the registered files so that city staff can easily open drawings based on the boundary extent chosen.NEXT POINTOther fields that are required in GIS but not provided in the digitally submitted AutoCAD files are filled out by city GIS staff or other automation processes.
  14. The benefits of the digital submission process and data compliancy checks address the main issues with the old method of capturing infrastructure data.NEXT POINTThe turn around time for GIS to actually capture the data has been drastically reduced. What might have taken a week to do has been reduced to a few hours.NEXT POINTAnd with the checkers we can ensure that the data being provided and loaded is valid and complete. No more multiple data entry. No more missing data. The data currently coming in is better and more complete than ever before.NEXT POINTAll parties understand what they need to do. The digitally submitted files must contain the needed information in the format specified. Data is checked based on the standards and rules specified, and issues are returned to the submitter for correction and resubmission. Approved data is loaded and additional data not required from the drawings are captured by GIS staff. GIS staff also remove any replaced features.NEXT POINTEveryone is happy. All data is available to city staff to view, report, and work on. GIS staff are not guestimating to fill in missing data. And sure, initially there was some grumbling from consultants about the additional work, but they are starting to realize that the amount of work is not that great, and they are benefitting as they in turn request this GIS data when planning new construction projects or performing any consulting work.
  15. What’s next? Continue improving.NEXT POINTThere are constant changes taking place in the GIS database because of new inspections, maintenance programs, and work order templates in Cityworks, not to mention just additional features or attributes that were not previously captured. With all these changes it has been a challenge to keep the AutoCAD template that consultants download current, as well as knowing which versions of the checkers settings to use to check the submitted drawings.NEXT POINTOf course, keep finding uses for FME. Using it to assist in the loading of the data is the next logical step. Maybe even assist in deleting the features that need to be removed.NEXT POINTCurrently the infrastructureprojects done through and for the Region of Waterloo inside the City of Kitchener are not in any digital submission format and therefore can’t be checked for data compliancy. However, Regional requests for city data has increased over the last few years for various reasons. Just like the consultants using city GIS data for their construction projects and analysis consulting, it is hoped that these projects and others like them will eventually encourage the Region to implement a similar digital submission process that would allow for data compliancy checks.NEXT POINTThe ability to customize the basic settings of the checkers allows for data compliancy checks in other areas of the city. For example our Gas Utility staff have recently moved over to AutoCAD to leverage the use of the digital submission process and these data compliancy checkers on their drawings. We are starting to received drawings from them and expect that by the end of the upcoming construction season most drawings will be submitted in the required format.NEXT POINTAnd lastly using FME Server. Currently we have only one FME desktop license and digital submission files are sent to GIS staff for running through the checkers. Having FME Server could give Engineering staff, or the consultants themselves, access to submit the files directly to the checkers, which would therefore reduce the time taken to check for data compliancy.