SlideShare une entreprise Scribd logo
1  sur  21
Télécharger pour lire hors ligne
Integrated Data Management




Managing Large Data Sets for Smart Meters

Jacques Roy, IBM
·Click to add text
jacquesr@us.ibm.com

                                            © 2011 IBM Corporation
Integrated Data Management



    Overview

    • How much data do smart meters generate?
    • Introducing Informix TimeSeries benefits
    • Some technical details
    • Proof points
    • Solution partners
    • Demo




2                                                © 2011 IBM Corporation
Integrated Data Management



           Utility-Scale Smart Meter Deployments, Plans &
                              Proposals*
                                                        September 2010




                                                                                                 Deployment for
                                                                                                 >50% of end-users

                                                                                                 Deployment for
                                                                                                 <50% of end-users


    *This map represents smart meter deployments,
    planned deployments, and proposals by investor-
    owned utilities and large public power utilities.
                                                                         © 2010 The Institute for Electric Efficiency
                                                                            http://www.edisonfoundation.net/IEE/

3                                                                                                   © 2011 IBM Corporation
Integrated Data Management



Example of Changing Storage Requirements
    Changing Workloads For 10 Million Smart Meters:
    Now – Each meter read once per month
    Very soon – Each meter read once every 15 minutes (2920X)
    Regulations – Need to keep data on line for 3 years (PUC) and, perhaps, save for 7 years

                           Data for a Utility In California         350.4B



# of Records
Per Year for
 10M meters


                                                    3.65B
                       120M

                                                                                  Frequency
                   Monthly Meter                Daily Meter     15 Minute Meter   of reads
                      Reads                       Reads             Reads
4                                                                                  © 2011 IBM Corporation
Integrated Data Management



     Keeping up with Smart Meter Data

    Large amounts of data causes problems in 2 areas:

    A) Storage management
       • Will get expensive and cumbersome to maintain

    • Query performance
       • Compliance Reports must be completed before the end of each day
       • Customer portal queries must be handled in a timely manner
       • Customer billing




5                                                                    © 2011 IBM Corporation
Integrated Data Management



       Introducing IBM Informix Relational Database
    • Why IBM Informix?
      • Low cost/Low administration
        • Manage thousands of servers with one database administrator
        • Many installations have no DBA’s


      • High performance
        • Insert 100’s of thousands of records per second
        • Analytic functions not available in other database products


      • High Availability
        • Scale up, scale out, and disaster recovery


      • Native time series data support
        • Load and analyze time series data


 “The idea was to run the project in three cycles, and use a different type of analysis in each cycle, to see if we
could draw any conclusions in terms of how to promote change in the way people view their energy
consumption. With help from the IBM Hursley team, we quickly found that Informix TimeSeries could deliver
spectacular results.” Clive Eisen, Chief Technology Officer at Hildebrand.
6                                                                                                   © 2011 IBM Corporation
Integrated Data Management



    Key Strengths of Informix TimeSeries
    •   Performance
        • Extremely fast data access
           • Data clustered and sorted by time on disk to reduce I/O
        • Handles operations hard or impossible to do in standard SQL
        • Continuous Real-Time and Batch Data Loaders

    •   Space Savings
        • Typically saves 50% space over standard relational layout

    •   Toolkit approach
        • Allows users to develop their own algorithms to run in the database
        • Algorithms running in the database leverage the buffer pool for speed

    •   Easier
        • Conceptually closer to how users think of time series


7                                                                                 © 2011 IBM Corporation
Integrated Data Management


Typical Relational Schema for Smart Meters Data
                            Index                 Smart_Meters
                                                     Table
                    meter_id        Time         KWH        Voltage          ColN
                        1         1-1-11 12:00    Value 1   Value 2   ……..   Value N
      Table Grows




                        2         1-1-11 12:00    Value 1   Value 2   ……..   Value N

                        3         1-1-11 12:00    Value 1   Value 2   ……..   Value N

                       …               …             …        …       ……..     …

                        1         1-1-11 12:15    Value 1   Value 2   ……..   Value N

                        2         1-1-11 12:15    Value 1   Value 2   ……..   Value N

                        3         1-1-11 12:15    Value 1   Value 2   ……..   Value N

                       …               …             …        …       ……..     …


                    • Each row contains exactly one record = billions of rows
                    • Additional indexes are required for efficient lookups
                    • Data is appended to the end of the table as it arrives
                    • Meter Id’s stored in every record
                    • No concept of a missing row
8                                                                                      © 2011 IBM Corporation
Integrated Data Management



    Same Table using an Informix TimeSeries Schema
                                           Smart_Meters
                                              Table
    meter_id                                            Series
         1         [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]

         2         [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]

         3         [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]

         4         [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]

         …                                                       …


                                            Table grows

    • Each row contains a growing set of records = one row per meter
    • Data append to a row rather than to the end of the table
    • Meter Ids not stored in individual records
    • Data is clustered by meter id and sorted by time on disk
    • Missing values take no disk space, missing interval reads take 2 bytes
9                                                                                                     © 2011 IBM Corporation
Integrated Data Management



     Informix TimeSeries: Key Concepts
      • DBSpace
       • A logical unit of storage used by the database server
       • Made up of one or more physical storage units called chunks

      • Containers
       • Specialized storage for TimeSeries in dbspaces

      • TimeSeries data element: row type
       • Flexibility to define as many parts as needed

      • TimeSeries types: regular, irregular
       • Covers regular intervals and sparse data distribution

      • Calendar
       • Defines business patterns

      • Relational view: VTI and processing return type

10                                                                     © 2011 IBM Corporation
Integrated Data Management



     Informix TimeSeries: Space savings

      • Example of simple meter record:
       {loc_id, timestamp, register_1, register_2, register_3}
        8 bytes 12 bytes 4 bytes       4 bytes     4bytes

      • TimeSeries savings: no duplication of loc_id, timestamp
       • Record size: 12 bytes instead of 32 bytes

      • Impact: 10M meters, 3 years, 15-minute interval
       • Savings: ~20TB
         20 bytes * 10M meters * 96 intervals/day * 3 years * 365 days/year

      • Index on meter records:
       • Relational: 1 trillion entries on loc_id, timestamp
       • TimeSeries: 10M entries on loc_id



11                                                                      © 2011 IBM Corporation
Integrated Data Management



     Informix TimeSeries: I/O advantages

      • Less data to read, less index entries to navigate


      • Meter data grouped on pages
       • Reading 1 day of data → One I/O (+ index I/O)
       • Relational: no grouping guarantees, could be 96 I/Os (+ index I/O)




12                                                                       © 2011 IBM Corporation
Integrated Data Management



     Informix TimeSeries: Processing advantages

      • TimeSeries data is accessed in time order
       • Relational model has no order guarantees – Sorting required

      • Library of SQL functions to process TimeSeries
       • 100+ functions
       • Ex: Aggregate data from 15-min interval to daily interval
             Calculate a moving average

      • Ability to write custom functions
       • Write business-specific processing
       • Streamline the processing instead of generic functions
         • Can greatly reduce code length → Significant performance
           improvements




13                                                                     © 2011 IBM Corporation
Integrated Data Management


     Informix TimeSeries vs Relational Database Systems
     Comparison of 1M Meter POC at Large US Energy Provider

                                      Informix              Competitor's Relational
1M Meter @ 15min POC                                                                    Improvement
                                      TimeSeries            Database System
Load Time                             18 Minutes            7 Hours                     23x faster
Report Generate Time                  Seconds to 11 min     2-7 HOURS                   38x faster
Storage Space                         350GB                 1.3 TB                      <1/3 storage


                   IBM Informix TimeSeries
                   Relational Database System                                             Based on actual
                                                                                        benchmark test run
                                                                                        for a large electrical
                                                                                           utility company
 Processing                                               Disk
    Time                                                  Storage
  (Hours)                                                 Space




            Data Loading      Reports                                 Relational   IBM Informix
                                                                      Database     TimeSeries


14                                                                                         © 2011 IBM Corporation
Integrated Data Management


         Informix TimeSeries vs Relational Database Systems
         Comparison of Published Benchmarks for Meter Data Management
                                                                           Daily           Total      Total                DB            App                 DB       App
                                                        Meters
                                                                           Reads           Cores      RAM                  cores         cores               RAM      RAM
      Informix TimeSeries                                                                                                                                             (shared
                                                        100M               4.9B            16         500                  16            (shared)            500
                                                                                                                                                                      )

      The Competition *                                 5.5M               1.06B           456        3668                 96            360                 768      2900


                         Daily Readings (meters * registers * intervals)                                                    Database Resources (CPU cores)




Informix TimeSeries                                                                4,900,000,000   Informix TimeSeries     16




     The Competition            1,061,500,000                                                         The Competition                                                  96




                       5 times the performance                                                                           < 1/5 the resources
      … with significantly simpler management using a single node system
     * Based on published Oracle benchmark
       http://www.oracle.com/us/industries/utilities/performance-benchmark-exadata-wp-161572.pdf
15                                                                                                                                                            © 2011 IBM Corporation
Integrated Data Management

      Informix TimeSeries – Success with leading MDM providers
http://smart-grid...                                    http://www.metering.com/node/20062




                                                       MDM offers scalability to 100 million smart meters




                 . “We feel that the Informix          “Performance testing results of
                 TimeSeries technology lends           AMT-SYBEX’s Affinity Meterflow
                 itself extremely well to the          meter data management (MDM)
                 performance characteristics, data     application using IBM Informix
                 types and high-volume                 TimeSeries software has
                 processing required for the next      demonstrated the capability to
                 generation of smart metering and      offer linear scalability up to 100
                 smart grid applications.”             million meters to load and
                 - David Hubbard, CTO and              process meter data at 30-minute
                 co-founder of Ecologic Analytics      intervals in less than 8 hours.”


 16                                                                                          © 2011 IBM Corporation
Integrated Data Management



     Client Success: Hildebrand

Hildebrand is a technology consultant company         on the Digital Environment Home Energy
Management System (DEHEMS) project. The Hildebrand team was asked by the UK government to find
a way to scale up its energy monitoring solution and enable it to monitor three million homes.


                         What’s Smart?
                         • Energy management thru real time monitoring
                         • Ability to monitor scales to over 3 million homes
                         • Reduced energy

                         Business Benefits
                         • Helping people make better decisions about energy efficiency in
                           the home.
                         • Collect, store and analyse up to • 50,000 data points per second
                         • Delivers high performance on low-cost hardware by leveraging
                           Informix time series data management technologies.




17                                                                               © 2011 IBM Corporation
Integrated Data Management



      Hildebrand: 3 Million Meters
 • Test involved 3,000,000 Homes                            • Hardware/Software Used
     • Data generated every 6 seconds!                        • Intel with 8 cores running:
                                                                • 64 bit SUSE Linux v10
 • Up to 26 values recorded                                     • 16 GB of memory
     • Meter ID                                               • Informix workgroup edition
     • Timestamp                                              • Hildebrand software
     • 3 Electricity phases
     • 1 Gas reading
     • 20 Individual electrical sockets in the house


 • Data collected every 6 seconds
     • Aggregated to per-minute readings
     • Minute readings bulk-loaded into Informix every 10
       minutes
     • Average load was about 50,000 inserts per second

18                                                                                  © 2011 IBM Corporation
Integrated Data Management



     Consumer Education




19                                        © 2011 IBM Corporation
Integrated Data Management



     AMT/SYBEX
     Proving the concept for smart metering data management
     • The Need:
       • Extend AMT/Sybex’s Data Transfer Solution to:                   “We owe a great deal to
         • Load, validate, store, and provide smart meter interval and   IBM, both for the Informix
           event data to external system                                 technology itself, and for the
                                                                         fantastic support from all
     • The Challenge:                                                    levels of the organisation.”
       • Process the enormous volume of data in a timely                      — Gordon Brown, DTS Product
         manner                                                                        Owner, AMT-SYBEX
       • Not to require a huge investment in new hardware
                                                                         Solution components:
     • The Solution:
                                                                          IBM® Informix® TimeSeries™
       • Combined effort with IBM Research and IBM Informix
       • Informix TimeSeries provides the core capabilities to            IBM Research
         store and process the meter and event data:
         •   VEE
         •   Real-time energy monitoring
         •   Analytics for developing new tariff rates
         •   Help to smooth peaks in demand




20                                                                                         © 2011 IBM Corporation
Integrated Data Management



     AMT/Sybex Tests and Performance Measurements
     The main tests for AMT-SYBEX were to prove key business functions at high
      volume:
         • Can high volumes of data be loaded into Smart DTS?
         • Can data be analyzed and processed in Smart DTS in a timely manner?

     To facilitate these requirements, the following tests were carried out:
         • These results are for 10 million meters storing data every ½ hour data for one day.

      Module                              Time                           Readings/intervals per
                                                                         second
      Technical Validation and            13 minutes                     >600,000
      Transformation
      High Speed TimeSeries               50 minutes                     >150,000
      database load (with full
      logging)
      Validation and Estimation           33 minutes                     >240,000



         Full processing end to end for 10 million meter points with half-hourly interval
         data on a single mid-range P-series 8 CPU server is just over 1 ½ hours

21                                                                                           © 2011 IBM Corporation

Contenu connexe

Similaire à Ugif 12 2011-smart meters-11102011

Why Gateways are Important in Your IoT Architecture
Why Gateways are Important in Your IoT ArchitectureWhy Gateways are Important in Your IoT Architecture
Why Gateways are Important in Your IoT ArchitectureIBM Analytics
 
Io t world_2016_iot_smart_gateways_moe
Io t world_2016_iot_smart_gateways_moeIo t world_2016_iot_smart_gateways_moe
Io t world_2016_iot_smart_gateways_moeShawn Moe
 
eMeter Interval Billing Application Data Sheet
eMeter Interval Billing Application Data SheeteMeter Interval Billing Application Data Sheet
eMeter Interval Billing Application Data SheetMichaline Todd
 
Invantive at Eusprig 2014
Invantive at Eusprig 2014Invantive at Eusprig 2014
Invantive at Eusprig 2014eusprig
 
Ibm_IoT_Architecture_and_Capabilities
Ibm_IoT_Architecture_and_CapabilitiesIbm_IoT_Architecture_and_Capabilities
Ibm_IoT_Architecture_and_CapabilitiesIBM_Info_Management
 
eMeter EnergyIP Data Sheet
eMeter EnergyIP Data SheeteMeter EnergyIP Data Sheet
eMeter EnergyIP Data SheetMichaline Todd
 
IBM Internet-of-Things architecture and capabilities
IBM Internet-of-Things architecture and capabilitiesIBM Internet-of-Things architecture and capabilities
IBM Internet-of-Things architecture and capabilitiesIBM_Info_Management
 
CA Nimsoft ecoMeter
CA Nimsoft ecoMeterCA Nimsoft ecoMeter
CA Nimsoft ecoMeterCA Nimsoft
 
Living objects network performance_management_v2
Living objects network performance_management_v2Living objects network performance_management_v2
Living objects network performance_management_v2Yoan SMADJA
 
Data Centre Compute and Overhead Costs - Delivering End-to-end KPIs
Data Centre Compute and Overhead Costs - Delivering End-to-end KPIsData Centre Compute and Overhead Costs - Delivering End-to-end KPIs
Data Centre Compute and Overhead Costs - Delivering End-to-end KPIsJISC's Green ICT Programme
 
Chapter 05- IT infrastucture and emerging technologies.pptx
Chapter 05- IT infrastucture and emerging technologies.pptxChapter 05- IT infrastucture and emerging technologies.pptx
Chapter 05- IT infrastucture and emerging technologies.pptxjack732582
 
Introduction to Database Management Systems
Introduction to Database Management SystemsIntroduction to Database Management Systems
Introduction to Database Management SystemsAdri Jovin
 
The CIBER / CA partnership & Why CIBER is moving to Nimsoft Monitor
The CIBER / CA partnership & Why CIBER is moving to Nimsoft MonitorThe CIBER / CA partnership & Why CIBER is moving to Nimsoft Monitor
The CIBER / CA partnership & Why CIBER is moving to Nimsoft Monitor CA Nimsoft
 
IBM IoT Architecture and Capabilities at the Edge and Cloud
IBM IoT Architecture and Capabilities at the Edge and Cloud IBM IoT Architecture and Capabilities at the Edge and Cloud
IBM IoT Architecture and Capabilities at the Edge and Cloud Pradeep Natarajan
 
UTILITY BILL PAYMENT BASED ON NETWORK SECURITY
UTILITY BILL PAYMENT BASED ON NETWORK SECURITYUTILITY BILL PAYMENT BASED ON NETWORK SECURITY
UTILITY BILL PAYMENT BASED ON NETWORK SECURITYIRJET Journal
 
BEMServer - open source platform for building energy management
BEMServer - open source platform for building energy managementBEMServer - open source platform for building energy management
BEMServer - open source platform for building energy managementpbourreau
 

Similaire à Ugif 12 2011-smart meters-11102011 (20)

Why Gateways are Important in Your IoT Architecture
Why Gateways are Important in Your IoT ArchitectureWhy Gateways are Important in Your IoT Architecture
Why Gateways are Important in Your IoT Architecture
 
Io t world_2016_iot_smart_gateways_moe
Io t world_2016_iot_smart_gateways_moeIo t world_2016_iot_smart_gateways_moe
Io t world_2016_iot_smart_gateways_moe
 
eMeter Interval Billing Application Data Sheet
eMeter Interval Billing Application Data SheeteMeter Interval Billing Application Data Sheet
eMeter Interval Billing Application Data Sheet
 
Invantive at Eusprig 2014
Invantive at Eusprig 2014Invantive at Eusprig 2014
Invantive at Eusprig 2014
 
Ibm_IoT_Architecture_and_Capabilities
Ibm_IoT_Architecture_and_CapabilitiesIbm_IoT_Architecture_and_Capabilities
Ibm_IoT_Architecture_and_Capabilities
 
eMeter EnergyIP Data Sheet
eMeter EnergyIP Data SheeteMeter EnergyIP Data Sheet
eMeter EnergyIP Data Sheet
 
Enertia Gen
Enertia GenEnertia Gen
Enertia Gen
 
DATA WAREHOUSING
DATA WAREHOUSINGDATA WAREHOUSING
DATA WAREHOUSING
 
IBM Internet-of-Things architecture and capabilities
IBM Internet-of-Things architecture and capabilitiesIBM Internet-of-Things architecture and capabilities
IBM Internet-of-Things architecture and capabilities
 
CA Nimsoft ecoMeter
CA Nimsoft ecoMeterCA Nimsoft ecoMeter
CA Nimsoft ecoMeter
 
Living objects network performance_management_v2
Living objects network performance_management_v2Living objects network performance_management_v2
Living objects network performance_management_v2
 
Data Centre Compute and Overhead Costs - Delivering End-to-end KPIs
Data Centre Compute and Overhead Costs - Delivering End-to-end KPIsData Centre Compute and Overhead Costs - Delivering End-to-end KPIs
Data Centre Compute and Overhead Costs - Delivering End-to-end KPIs
 
I learning lot
I learning lotI learning lot
I learning lot
 
Chapter 05- IT infrastucture and emerging technologies.pptx
Chapter 05- IT infrastucture and emerging technologies.pptxChapter 05- IT infrastucture and emerging technologies.pptx
Chapter 05- IT infrastucture and emerging technologies.pptx
 
Introduction to Database Management Systems
Introduction to Database Management SystemsIntroduction to Database Management Systems
Introduction to Database Management Systems
 
The CIBER / CA partnership & Why CIBER is moving to Nimsoft Monitor
The CIBER / CA partnership & Why CIBER is moving to Nimsoft MonitorThe CIBER / CA partnership & Why CIBER is moving to Nimsoft Monitor
The CIBER / CA partnership & Why CIBER is moving to Nimsoft Monitor
 
Final mis power
Final mis powerFinal mis power
Final mis power
 
IBM IoT Architecture and Capabilities at the Edge and Cloud
IBM IoT Architecture and Capabilities at the Edge and Cloud IBM IoT Architecture and Capabilities at the Edge and Cloud
IBM IoT Architecture and Capabilities at the Edge and Cloud
 
UTILITY BILL PAYMENT BASED ON NETWORK SECURITY
UTILITY BILL PAYMENT BASED ON NETWORK SECURITYUTILITY BILL PAYMENT BASED ON NETWORK SECURITY
UTILITY BILL PAYMENT BASED ON NETWORK SECURITY
 
BEMServer - open source platform for building energy management
BEMServer - open source platform for building energy managementBEMServer - open source platform for building energy management
BEMServer - open source platform for building energy management
 

Plus de UGIF

UGIF 09 2013 Fy13 q3, corporate presentation the inflection point in the ap...
UGIF 09 2013 Fy13 q3, corporate presentation   the inflection point in the ap...UGIF 09 2013 Fy13 q3, corporate presentation   the inflection point in the ap...
UGIF 09 2013 Fy13 q3, corporate presentation the inflection point in the ap...UGIF
 
Ugif 09 2013 open source - session tech
Ugif 09 2013   open source - session techUgif 09 2013   open source - session tech
Ugif 09 2013 open source - session techUGIF
 
Ugif 09 2013 new environment and dynamic setting in ids 12.10
Ugif 09 2013   new environment and dynamic setting in ids 12.10Ugif 09 2013   new environment and dynamic setting in ids 12.10
Ugif 09 2013 new environment and dynamic setting in ids 12.10UGIF
 
Ugif 09 2013 open source
Ugif 09 2013   open sourceUgif 09 2013   open source
Ugif 09 2013 open sourceUGIF
 
Ugif 09 2013
Ugif 09 2013Ugif 09 2013
Ugif 09 2013UGIF
 
Ugif 09 2013 psm
Ugif 09 2013   psmUgif 09 2013   psm
Ugif 09 2013 psmUGIF
 
Ugif 09 2013 friug 201309 axional web studio
Ugif 09 2013 friug 201309   axional web studioUgif 09 2013 friug 201309   axional web studio
Ugif 09 2013 friug 201309 axional web studioUGIF
 
Ugif 10 2012 ppt0000001
Ugif 10 2012 ppt0000001Ugif 10 2012 ppt0000001
Ugif 10 2012 ppt0000001UGIF
 
Ugif 10 2012 informix pssc-benchmark -l.revel_oct2012
Ugif 10 2012 informix pssc-benchmark -l.revel_oct2012Ugif 10 2012 informix pssc-benchmark -l.revel_oct2012
Ugif 10 2012 informix pssc-benchmark -l.revel_oct2012UGIF
 
Ugif 10 2012 beauty ofifmxdiskstructs ugif
Ugif 10 2012 beauty ofifmxdiskstructs ugifUgif 10 2012 beauty ofifmxdiskstructs ugif
Ugif 10 2012 beauty ofifmxdiskstructs ugifUGIF
 
Ugif 10 2012 lycia2 introduction in 45 minutes
Ugif 10 2012 lycia2 introduction in 45 minutesUgif 10 2012 lycia2 introduction in 45 minutes
Ugif 10 2012 lycia2 introduction in 45 minutesUGIF
 
Ugif 10 2012 genero ugif october 3, 2012 ibm france, français
Ugif 10 2012 genero   ugif october 3, 2012  ibm france, français Ugif 10 2012 genero   ugif october 3, 2012  ibm france, français
Ugif 10 2012 genero ugif october 3, 2012 ibm france, français UGIF
 
Ugif 10 2012 iiug paris-business-update
Ugif 10 2012 iiug paris-business-updateUgif 10 2012 iiug paris-business-update
Ugif 10 2012 iiug paris-business-updateUGIF
 
Ugif 10 2012 ppt0000002
Ugif 10 2012 ppt0000002Ugif 10 2012 ppt0000002
Ugif 10 2012 ppt0000002UGIF
 
Ugif 12 2011-informix iwa
Ugif 12 2011-informix iwaUgif 12 2011-informix iwa
Ugif 12 2011-informix iwaUGIF
 
Ugif 12 2011-ibm cap-seine
Ugif 12 2011-ibm cap-seineUgif 12 2011-ibm cap-seine
Ugif 12 2011-ibm cap-seineUGIF
 
Ugif 12 2011-france ug12142011-tech_ts
Ugif 12 2011-france ug12142011-tech_tsUgif 12 2011-france ug12142011-tech_ts
Ugif 12 2011-france ug12142011-tech_tsUGIF
 
Ugif 12 2011-four js primer presentation - new graphic charter - short versio...
Ugif 12 2011-four js primer presentation - new graphic charter - short versio...Ugif 12 2011-four js primer presentation - new graphic charter - short versio...
Ugif 12 2011-four js primer presentation - new graphic charter - short versio...UGIF
 
Ugif 12 2011-discover informix keynote 2012
Ugif 12 2011-discover informix keynote 2012Ugif 12 2011-discover informix keynote 2012
Ugif 12 2011-discover informix keynote 2012UGIF
 
Ugif 04 2011 storage prov-pot_march_2011
Ugif 04 2011   storage prov-pot_march_2011Ugif 04 2011   storage prov-pot_march_2011
Ugif 04 2011 storage prov-pot_march_2011UGIF
 

Plus de UGIF (20)

UGIF 09 2013 Fy13 q3, corporate presentation the inflection point in the ap...
UGIF 09 2013 Fy13 q3, corporate presentation   the inflection point in the ap...UGIF 09 2013 Fy13 q3, corporate presentation   the inflection point in the ap...
UGIF 09 2013 Fy13 q3, corporate presentation the inflection point in the ap...
 
Ugif 09 2013 open source - session tech
Ugif 09 2013   open source - session techUgif 09 2013   open source - session tech
Ugif 09 2013 open source - session tech
 
Ugif 09 2013 new environment and dynamic setting in ids 12.10
Ugif 09 2013   new environment and dynamic setting in ids 12.10Ugif 09 2013   new environment and dynamic setting in ids 12.10
Ugif 09 2013 new environment and dynamic setting in ids 12.10
 
Ugif 09 2013 open source
Ugif 09 2013   open sourceUgif 09 2013   open source
Ugif 09 2013 open source
 
Ugif 09 2013
Ugif 09 2013Ugif 09 2013
Ugif 09 2013
 
Ugif 09 2013 psm
Ugif 09 2013   psmUgif 09 2013   psm
Ugif 09 2013 psm
 
Ugif 09 2013 friug 201309 axional web studio
Ugif 09 2013 friug 201309   axional web studioUgif 09 2013 friug 201309   axional web studio
Ugif 09 2013 friug 201309 axional web studio
 
Ugif 10 2012 ppt0000001
Ugif 10 2012 ppt0000001Ugif 10 2012 ppt0000001
Ugif 10 2012 ppt0000001
 
Ugif 10 2012 informix pssc-benchmark -l.revel_oct2012
Ugif 10 2012 informix pssc-benchmark -l.revel_oct2012Ugif 10 2012 informix pssc-benchmark -l.revel_oct2012
Ugif 10 2012 informix pssc-benchmark -l.revel_oct2012
 
Ugif 10 2012 beauty ofifmxdiskstructs ugif
Ugif 10 2012 beauty ofifmxdiskstructs ugifUgif 10 2012 beauty ofifmxdiskstructs ugif
Ugif 10 2012 beauty ofifmxdiskstructs ugif
 
Ugif 10 2012 lycia2 introduction in 45 minutes
Ugif 10 2012 lycia2 introduction in 45 minutesUgif 10 2012 lycia2 introduction in 45 minutes
Ugif 10 2012 lycia2 introduction in 45 minutes
 
Ugif 10 2012 genero ugif october 3, 2012 ibm france, français
Ugif 10 2012 genero   ugif october 3, 2012  ibm france, français Ugif 10 2012 genero   ugif october 3, 2012  ibm france, français
Ugif 10 2012 genero ugif october 3, 2012 ibm france, français
 
Ugif 10 2012 iiug paris-business-update
Ugif 10 2012 iiug paris-business-updateUgif 10 2012 iiug paris-business-update
Ugif 10 2012 iiug paris-business-update
 
Ugif 10 2012 ppt0000002
Ugif 10 2012 ppt0000002Ugif 10 2012 ppt0000002
Ugif 10 2012 ppt0000002
 
Ugif 12 2011-informix iwa
Ugif 12 2011-informix iwaUgif 12 2011-informix iwa
Ugif 12 2011-informix iwa
 
Ugif 12 2011-ibm cap-seine
Ugif 12 2011-ibm cap-seineUgif 12 2011-ibm cap-seine
Ugif 12 2011-ibm cap-seine
 
Ugif 12 2011-france ug12142011-tech_ts
Ugif 12 2011-france ug12142011-tech_tsUgif 12 2011-france ug12142011-tech_ts
Ugif 12 2011-france ug12142011-tech_ts
 
Ugif 12 2011-four js primer presentation - new graphic charter - short versio...
Ugif 12 2011-four js primer presentation - new graphic charter - short versio...Ugif 12 2011-four js primer presentation - new graphic charter - short versio...
Ugif 12 2011-four js primer presentation - new graphic charter - short versio...
 
Ugif 12 2011-discover informix keynote 2012
Ugif 12 2011-discover informix keynote 2012Ugif 12 2011-discover informix keynote 2012
Ugif 12 2011-discover informix keynote 2012
 
Ugif 04 2011 storage prov-pot_march_2011
Ugif 04 2011   storage prov-pot_march_2011Ugif 04 2011   storage prov-pot_march_2011
Ugif 04 2011 storage prov-pot_march_2011
 

Dernier

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceMartin Humpolec
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 

Dernier (20)

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 

Ugif 12 2011-smart meters-11102011

  • 1. Integrated Data Management Managing Large Data Sets for Smart Meters Jacques Roy, IBM ·Click to add text jacquesr@us.ibm.com © 2011 IBM Corporation
  • 2. Integrated Data Management Overview • How much data do smart meters generate? • Introducing Informix TimeSeries benefits • Some technical details • Proof points • Solution partners • Demo 2 © 2011 IBM Corporation
  • 3. Integrated Data Management Utility-Scale Smart Meter Deployments, Plans & Proposals* September 2010 Deployment for >50% of end-users Deployment for <50% of end-users *This map represents smart meter deployments, planned deployments, and proposals by investor- owned utilities and large public power utilities. © 2010 The Institute for Electric Efficiency http://www.edisonfoundation.net/IEE/ 3 © 2011 IBM Corporation
  • 4. Integrated Data Management Example of Changing Storage Requirements Changing Workloads For 10 Million Smart Meters: Now – Each meter read once per month Very soon – Each meter read once every 15 minutes (2920X) Regulations – Need to keep data on line for 3 years (PUC) and, perhaps, save for 7 years Data for a Utility In California 350.4B # of Records Per Year for 10M meters 3.65B 120M Frequency Monthly Meter Daily Meter 15 Minute Meter of reads Reads Reads Reads 4 © 2011 IBM Corporation
  • 5. Integrated Data Management Keeping up with Smart Meter Data Large amounts of data causes problems in 2 areas: A) Storage management • Will get expensive and cumbersome to maintain • Query performance • Compliance Reports must be completed before the end of each day • Customer portal queries must be handled in a timely manner • Customer billing 5 © 2011 IBM Corporation
  • 6. Integrated Data Management Introducing IBM Informix Relational Database • Why IBM Informix? • Low cost/Low administration • Manage thousands of servers with one database administrator • Many installations have no DBA’s • High performance • Insert 100’s of thousands of records per second • Analytic functions not available in other database products • High Availability • Scale up, scale out, and disaster recovery • Native time series data support • Load and analyze time series data “The idea was to run the project in three cycles, and use a different type of analysis in each cycle, to see if we could draw any conclusions in terms of how to promote change in the way people view their energy consumption. With help from the IBM Hursley team, we quickly found that Informix TimeSeries could deliver spectacular results.” Clive Eisen, Chief Technology Officer at Hildebrand. 6 © 2011 IBM Corporation
  • 7. Integrated Data Management Key Strengths of Informix TimeSeries • Performance • Extremely fast data access • Data clustered and sorted by time on disk to reduce I/O • Handles operations hard or impossible to do in standard SQL • Continuous Real-Time and Batch Data Loaders • Space Savings • Typically saves 50% space over standard relational layout • Toolkit approach • Allows users to develop their own algorithms to run in the database • Algorithms running in the database leverage the buffer pool for speed • Easier • Conceptually closer to how users think of time series 7 © 2011 IBM Corporation
  • 8. Integrated Data Management Typical Relational Schema for Smart Meters Data Index Smart_Meters Table meter_id Time KWH Voltage ColN 1 1-1-11 12:00 Value 1 Value 2 …….. Value N Table Grows 2 1-1-11 12:00 Value 1 Value 2 …….. Value N 3 1-1-11 12:00 Value 1 Value 2 …….. Value N … … … … …….. … 1 1-1-11 12:15 Value 1 Value 2 …….. Value N 2 1-1-11 12:15 Value 1 Value 2 …….. Value N 3 1-1-11 12:15 Value 1 Value 2 …….. Value N … … … … …….. … • Each row contains exactly one record = billions of rows • Additional indexes are required for efficient lookups • Data is appended to the end of the table as it arrives • Meter Id’s stored in every record • No concept of a missing row 8 © 2011 IBM Corporation
  • 9. Integrated Data Management Same Table using an Informix TimeSeries Schema Smart_Meters Table meter_id Series 1 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] 2 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] 3 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] 4 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] … … Table grows • Each row contains a growing set of records = one row per meter • Data append to a row rather than to the end of the table • Meter Ids not stored in individual records • Data is clustered by meter id and sorted by time on disk • Missing values take no disk space, missing interval reads take 2 bytes 9 © 2011 IBM Corporation
  • 10. Integrated Data Management Informix TimeSeries: Key Concepts • DBSpace • A logical unit of storage used by the database server • Made up of one or more physical storage units called chunks • Containers • Specialized storage for TimeSeries in dbspaces • TimeSeries data element: row type • Flexibility to define as many parts as needed • TimeSeries types: regular, irregular • Covers regular intervals and sparse data distribution • Calendar • Defines business patterns • Relational view: VTI and processing return type 10 © 2011 IBM Corporation
  • 11. Integrated Data Management Informix TimeSeries: Space savings • Example of simple meter record: {loc_id, timestamp, register_1, register_2, register_3} 8 bytes 12 bytes 4 bytes 4 bytes 4bytes • TimeSeries savings: no duplication of loc_id, timestamp • Record size: 12 bytes instead of 32 bytes • Impact: 10M meters, 3 years, 15-minute interval • Savings: ~20TB 20 bytes * 10M meters * 96 intervals/day * 3 years * 365 days/year • Index on meter records: • Relational: 1 trillion entries on loc_id, timestamp • TimeSeries: 10M entries on loc_id 11 © 2011 IBM Corporation
  • 12. Integrated Data Management Informix TimeSeries: I/O advantages • Less data to read, less index entries to navigate • Meter data grouped on pages • Reading 1 day of data → One I/O (+ index I/O) • Relational: no grouping guarantees, could be 96 I/Os (+ index I/O) 12 © 2011 IBM Corporation
  • 13. Integrated Data Management Informix TimeSeries: Processing advantages • TimeSeries data is accessed in time order • Relational model has no order guarantees – Sorting required • Library of SQL functions to process TimeSeries • 100+ functions • Ex: Aggregate data from 15-min interval to daily interval Calculate a moving average • Ability to write custom functions • Write business-specific processing • Streamline the processing instead of generic functions • Can greatly reduce code length → Significant performance improvements 13 © 2011 IBM Corporation
  • 14. Integrated Data Management Informix TimeSeries vs Relational Database Systems Comparison of 1M Meter POC at Large US Energy Provider Informix Competitor's Relational 1M Meter @ 15min POC Improvement TimeSeries Database System Load Time 18 Minutes 7 Hours 23x faster Report Generate Time Seconds to 11 min 2-7 HOURS 38x faster Storage Space 350GB 1.3 TB <1/3 storage IBM Informix TimeSeries Relational Database System Based on actual benchmark test run for a large electrical utility company Processing Disk Time Storage (Hours) Space Data Loading Reports Relational IBM Informix Database TimeSeries 14 © 2011 IBM Corporation
  • 15. Integrated Data Management Informix TimeSeries vs Relational Database Systems Comparison of Published Benchmarks for Meter Data Management Daily Total Total DB App DB App Meters Reads Cores RAM cores cores RAM RAM Informix TimeSeries (shared 100M 4.9B 16 500 16 (shared) 500 ) The Competition * 5.5M 1.06B 456 3668 96 360 768 2900 Daily Readings (meters * registers * intervals) Database Resources (CPU cores) Informix TimeSeries 4,900,000,000 Informix TimeSeries 16 The Competition 1,061,500,000 The Competition 96 5 times the performance < 1/5 the resources … with significantly simpler management using a single node system * Based on published Oracle benchmark http://www.oracle.com/us/industries/utilities/performance-benchmark-exadata-wp-161572.pdf 15 © 2011 IBM Corporation
  • 16. Integrated Data Management Informix TimeSeries – Success with leading MDM providers http://smart-grid... http://www.metering.com/node/20062 MDM offers scalability to 100 million smart meters . “We feel that the Informix “Performance testing results of TimeSeries technology lends AMT-SYBEX’s Affinity Meterflow itself extremely well to the meter data management (MDM) performance characteristics, data application using IBM Informix types and high-volume TimeSeries software has processing required for the next demonstrated the capability to generation of smart metering and offer linear scalability up to 100 smart grid applications.” million meters to load and - David Hubbard, CTO and process meter data at 30-minute co-founder of Ecologic Analytics intervals in less than 8 hours.” 16 © 2011 IBM Corporation
  • 17. Integrated Data Management Client Success: Hildebrand Hildebrand is a technology consultant company on the Digital Environment Home Energy Management System (DEHEMS) project. The Hildebrand team was asked by the UK government to find a way to scale up its energy monitoring solution and enable it to monitor three million homes. What’s Smart? • Energy management thru real time monitoring • Ability to monitor scales to over 3 million homes • Reduced energy Business Benefits • Helping people make better decisions about energy efficiency in the home. • Collect, store and analyse up to • 50,000 data points per second • Delivers high performance on low-cost hardware by leveraging Informix time series data management technologies. 17 © 2011 IBM Corporation
  • 18. Integrated Data Management Hildebrand: 3 Million Meters • Test involved 3,000,000 Homes • Hardware/Software Used • Data generated every 6 seconds! • Intel with 8 cores running: • 64 bit SUSE Linux v10 • Up to 26 values recorded • 16 GB of memory • Meter ID • Informix workgroup edition • Timestamp • Hildebrand software • 3 Electricity phases • 1 Gas reading • 20 Individual electrical sockets in the house • Data collected every 6 seconds • Aggregated to per-minute readings • Minute readings bulk-loaded into Informix every 10 minutes • Average load was about 50,000 inserts per second 18 © 2011 IBM Corporation
  • 19. Integrated Data Management Consumer Education 19 © 2011 IBM Corporation
  • 20. Integrated Data Management AMT/SYBEX Proving the concept for smart metering data management • The Need: • Extend AMT/Sybex’s Data Transfer Solution to: “We owe a great deal to • Load, validate, store, and provide smart meter interval and IBM, both for the Informix event data to external system technology itself, and for the fantastic support from all • The Challenge: levels of the organisation.” • Process the enormous volume of data in a timely — Gordon Brown, DTS Product manner Owner, AMT-SYBEX • Not to require a huge investment in new hardware Solution components: • The Solution:  IBM® Informix® TimeSeries™ • Combined effort with IBM Research and IBM Informix • Informix TimeSeries provides the core capabilities to  IBM Research store and process the meter and event data: • VEE • Real-time energy monitoring • Analytics for developing new tariff rates • Help to smooth peaks in demand 20 © 2011 IBM Corporation
  • 21. Integrated Data Management AMT/Sybex Tests and Performance Measurements The main tests for AMT-SYBEX were to prove key business functions at high volume: • Can high volumes of data be loaded into Smart DTS? • Can data be analyzed and processed in Smart DTS in a timely manner? To facilitate these requirements, the following tests were carried out: • These results are for 10 million meters storing data every ½ hour data for one day. Module Time Readings/intervals per second Technical Validation and 13 minutes >600,000 Transformation High Speed TimeSeries 50 minutes >150,000 database load (with full logging) Validation and Estimation 33 minutes >240,000 Full processing end to end for 10 million meter points with half-hourly interval data on a single mid-range P-series 8 CPU server is just over 1 ½ hours 21 © 2011 IBM Corporation