The document discusses the Australian Bureau of Statistics' use of the Statistical Data and Metadata Exchange (SDMX) standard. It provides an overview of ABS, its principles and clients. It then describes two case studies of using SDMX: to provide national accounts data to the OECD, and detailed labor force data to the Reserve Bank of Australia. The benefits of SDMX include efficient data transfer and handling metadata and structure changes. Moving forward, ABS's vision is to use SDMX as the primary dissemination format for aggregate and time series data, both nationally and internationally. Key lessons and issues discussed are the costs of implementation and ensuring SDMX meets the needs of different user types.
1. SDMX @ ABS:SDMX @ ABS:
Why and How we use SDMXWhy and How we use SDMX
Graeme OakleyGraeme Oakley
Australian Bureau of StatisticsAustralian Bureau of Statistics
"Towards Implementation of SDMX - Jan 9-11, 2007"
2. Content of PresentationContent of Presentation
Introduction - objectives, overview of ABSIntroduction - objectives, overview of ABS
Principles relevant to ABS data and metadataPrinciples relevant to ABS data and metadata
disseminationdissemination
Case studies - one national, oneCase studies - one national, one
internationalinternational
Vision, Lessons, IssuesVision, Lessons, Issues
"Towards Implementation of SDMX - Jan 9-11, 2007"
3. "Towards Implementation of SDMX - Jan 9-11, 2007"
Objectives of PresentationObjectives of Presentation
Why SDMX fits with ABS philosophyWhy SDMX fits with ABS philosophy
concerning principles and standardsconcerning principles and standards
Tell our stories about a national andTell our stories about a national and
international experience using SDMXinternational experience using SDMX
Discuss the future vision, concerns andDiscuss the future vision, concerns and
lessonslessons
4. Australian Bureau of StatisticsAustralian Bureau of Statistics
Australia's national statistical agency, statisticalAustralia's national statistical agency, statistical
leadership role in National Statistical Serviceleadership role in National Statistical Service
ABS MissionABS Mission
".. assist and encourage informed decision-making, research".. assist and encourage informed decision-making, research
and discussion .."and discussion .."
A broad range of statistics on Australian societyA broad range of statistics on Australian society
Economic, Demographic, Social, Health, Tourism,Economic, Demographic, Social, Health, Tourism,
Culture and Leisure, Environment, ...Culture and Leisure, Environment, ...
Central and 8 Regional offices, about 2750 staff (FTE)Central and 8 Regional offices, about 2750 staff (FTE)
plus 800 interviewersplus 800 interviewers
"Towards Implementation of SDMX - Jan 9-11, 2007"
5. ABS PrinciplesABS Principles
RelevanceRelevance: good planning with a good: good planning with a good
understanding of current and future needs ofunderstanding of current and future needs of
users;users; timely and relatable statisticstimely and relatable statistics
IntegrityIntegrity: analysis and interpretation is objective;: analysis and interpretation is objective;
always publish our statistics;always publish our statistics; explain and informexplain and inform;;
methodologies and approaches open to scrutinymethodologies and approaches open to scrutiny
Access for allAccess for all: strive for accurate and widespread: strive for accurate and widespread
media coverage;media coverage; access through range of productsaccess through range of products
and servicesand services
"Towards Implementation of SDMX - Jan 9-11, 2007"
6. ABS Principles (ctd)ABS Principles (ctd)
ProfessionalismProfessionalism: exercise highest professional: exercise highest professional
standards; staff highly trained;standards; staff highly trained; development ofdevelopment of
statistical standardsstatistical standards; open about the; open about the quality of ourquality of our
statisticsstatistics
Trust of providersTrust of providers: encouraged to provide accurate: encouraged to provide accurate
information;information; data is strictly protecteddata is strictly protected; explain why; explain why
information is collected; every ABS officer requiredinformation is collected; every ABS officer required
to give an undertaking of secrecyto give an undertaking of secrecy
Source: ABS Corporate PlanSource: ABS Corporate Plan
"Towards Implementation of SDMX - Jan 9-11, 2007"
7. ABS - ClientsABS - Clients
Australia's national statistical agency serving:Australia's national statistical agency serving:
Australian GovernmentAustralian Government
State/Territory GovernmentsState/Territory Governments
Local GovernmentsLocal Governments
International statistical organisationsInternational statistical organisations
Business communityBusiness community
Education, religious, welfare, etc organisationsEducation, religious, welfare, etc organisations
MediaMedia
Individual members of societyIndividual members of society
"Towards Implementation of SDMX - Jan 9-11, 2007"
8. Principles and Standards ApproachPrinciples and Standards Approach
Data ManagementData Management - data is visible, accessible,- data is visible, accessible,
understandable, reliable, relatable, presentableunderstandable, reliable, relatable, presentable
MetadataMetadata - definitions (understanding), discovery,- definitions (understanding), discovery,
quality ('fitness for purpose')quality ('fitness for purpose')
ABS Enterprise ArchitectureABS Enterprise Architecture - follow mainstream- follow mainstream
standards, collaborate & partner where effective,standards, collaborate & partner where effective,
promote development of low complexity &promote development of low complexity &
maintainable solutions, remove dependence on vendormaintainable solutions, remove dependence on vendor
formats, extensible and scalable systems, use emergingformats, extensible and scalable systems, use emerging
technology as appropriatetechnology as appropriate
"Towards Implementation of SDMX - Jan 9-11, 2007"
9. Standards (ctd)Standards (ctd)
"Towards Implementation of SDMX - Jan 9-11, 2007"
DisseminationDissemination --
all publishable data in ABS Information Warehouseall publishable data in ABS Information Warehouse
metadata associated with productsmetadata associated with products
web site is prime dissemination channelweb site is prime dissemination channel
clients can 'pull' data for freeclients can 'pull' data for free
data stored once but published in many commondata stored once but published in many common
formatsformats
StatisticalStatistical - use international standards eg ICD,- use international standards eg ICD,
SITC; and frameworks eg SNASITC; and frameworks eg SNA
10. Case StudiesCase Studies
InternationalInternational - NAWWE with OECD (Note:- NAWWE with OECD (Note:
ABS responds to many information requests fromABS responds to many information requests from
international organisations, in variety of formats -international organisations, in variety of formats -
we are seeking an efficient solution that aligns withwe are seeking an efficient solution that aligns with
our principles and standards)our principles and standards)
NationalNational - Reserve Bank of Australia (RBA)- Reserve Bank of Australia (RBA)
[A large, sophisticated and important user of ABS[A large, sophisticated and important user of ABS
data. There are a number of other users of thisdata. There are a number of other users of this
type eg Treasury.]type eg Treasury.]
"Towards Implementation of SDMX - Jan 9-11, 2007"
11. International - OECD NAWWEInternational - OECD NAWWE
Involvement in pilot project since 2002/03Involvement in pilot project since 2002/03
Tasks include:Tasks include:
Set up metadata and load National Accounts data toSet up metadata and load National Accounts data to
ABS Information Warehouse to suit agreedABS Information Warehouse to suit agreed
NAWWE Data Structure DefinitionNAWWE Data Structure Definition
Extract from ABSIW and transform to agreedExtract from ABSIW and transform to agreed
SDMX message structure (XML format)SDMX message structure (XML format)
Place on ABS website (recently decommissionedPlace on ABS website (recently decommissioned
web page for original case study and yet to put upweb page for original case study and yet to put up
one for current study)one for current study)
"Towards Implementation of SDMX - Jan 9-11, 2007"
12. OECD
FAME Time Series
DB
ABS Information
Warehouse
ABS
Statistical
Products
EXCEL File
Hand entered
Current Process
"Towards Implementation of SDMX - Jan 9-11, 2007"
13. OECD
FAME Time
Series DB
ABS Information
Warehouse
ABS
Statistical
Products
Time Series XML
Transform to
SDMX
Publish to
www.abs.gov.au
Future Process
"Towards Implementation of SDMX - Jan 9-11, 2007"
14. ABS ExperienceABS Experience
Changed business process requiredChanged business process required
Although supporting the initiative, ABS NationalAlthough supporting the initiative, ABS National
Accounts group could not always provideAccounts group could not always provide
resources to do necessary setup workresources to do necessary setup work
Long periods on inactivity whilst OECD workedLong periods on inactivity whilst OECD worked
on new Data Structure Definitionon new Data Structure Definition
Experimental SDMX V1 files were deliveredExperimental SDMX V1 files were delivered
Now have SDMX V2 DSD to understand andNow have SDMX V2 DSD to understand and
implementimplement
"Towards Implementation of SDMX - Jan 9-11, 2007"
15. National - RBANational - RBA
"Towards Implementation of SDMX - Jan 9-11, 2007"
Central Bank for Australia (RBA) - receivesCentral Bank for Australia (RBA) - receives
mainly time series data from many sourcesmainly time series data from many sources
Many formats and different frequenciesMany formats and different frequencies
Previously maintained their time series data inPreviously maintained their time series data in
an EXCEL based spreadsheet systeman EXCEL based spreadsheet system
In past year, developed new system toIn past year, developed new system to
centralise time series data storage andcentralise time series data storage and
retrieval, based on SQL serverretrieval, based on SQL server
16. National - RBA (ctd)National - RBA (ctd)
ABS provided detailed Labour Force data inABS provided detailed Labour Force data in
SDMX V1 format as a trial (RBA currently takesSDMX V1 format as a trial (RBA currently takes
LFS data in a large number of EXCEL files andLFS data in a large number of EXCEL files and
some multidimensional cubes)some multidimensional cubes)
The RBA's new system has been demonstratedThe RBA's new system has been demonstrated
to internal business areas, and acceptedto internal business areas, and accepted
RBA want to extend use of SDMX to other ABSRBA want to extend use of SDMX to other ABS
series and improve the timelinessseries and improve the timeliness
"Towards Implementation of SDMX - Jan 9-11, 2007"
17. ABS RBALabour Force
data in SDMX
format
SQL Server DB
holding time
series data
XSLT
SQL Server
Integration
Services
(SSIS)
Email with file attached
What RBA do with SDMX files
"Towards Implementation of SDMX - Jan 9-11, 2007"
RBA
economists
18. What RBA see as benefits?What RBA see as benefits?
Efficient data transfer - reduces need for clericalEfficient data transfer - reduces need for clerical
intervention when dealing with spreadsheets, canintervention when dealing with spreadsheets, can
easily manipulate with programseasily manipulate with programs
Metadata rich - SDMX allows more metadata toMetadata rich - SDMX allows more metadata to
be conveyed compared to spreadsheetsbe conveyed compared to spreadsheets
Able to handle changes in time series metadataAble to handle changes in time series metadata
and data more easily - can programmatically checkand data more easily - can programmatically check
metadata for changesmetadata for changes
"Towards Implementation of SDMX - Jan 9-11, 2007"
19. What does RBA want ABS to change?What does RBA want ABS to change?
"Towards Implementation of SDMX - Jan 9-11, 2007"
Nothing related to the transfer format - SDMXNothing related to the transfer format - SDMX
Want the SDMX files on ABS web site so canWant the SDMX files on ABS web site so can
download at 11.30am (embargo time) ratherdownload at 11.30am (embargo time) rather
than wait for emailthan wait for email
Want ABS to extend use of SDMX to otherWant ABS to extend use of SDMX to other
statistical seriesstatistical series
Additional information related to identification ofAdditional information related to identification of
discontinued series, code setsdiscontinued series, code sets
20. VisionVision
"Towards Implementation of SDMX - Jan 9-11, 2007"
Vision - framework and standard that ABS can useVision - framework and standard that ABS can use
for all our international reporting; and also to usefor all our international reporting; and also to use
with large major national users of ABS data.with large major national users of ABS data.
SDMX is suitable as the primary disseminationSDMX is suitable as the primary dissemination
vehicle for machine readable aggregate and timevehicle for machine readable aggregate and time
series datasets for both national and internationalseries datasets for both national and international
clients. The SDMX format files will become ABSclients. The SDMX format files will become ABS
data products held on the web site for free access.data products held on the web site for free access.
21. LessonsLessons
SDMX has the potential to provide integrated dataSDMX has the potential to provide integrated data
exchange across organisational boundaries.exchange across organisational boundaries.
Challenge is not the SDMX 'technology', butChallenge is not the SDMX 'technology', but
dealing with 'meaning' and 'mapping' issuesdealing with 'meaning' and 'mapping' issues
Recognise costs involved in implementation,Recognise costs involved in implementation,
especially mapping local metadata and dataespecially mapping local metadata and data
structures to international requirement in eachstructures to international requirement in each
subject domain, and the importance of engagingsubject domain, and the importance of engaging
local subject matter expertslocal subject matter experts
"Towards Implementation of SDMX - Jan 9-11, 2007"
22. IssuesIssues
"Towards Implementation of SDMX - Jan 9-11, 2007"
Use SDMX for wide range of users, from technicallyUse SDMX for wide range of users, from technically
competent to small data integrator. A short shallowcompetent to small data integrator. A short shallow
learning curve with plenty of support, such as easylearning curve with plenty of support, such as easy
documentation, examples, tutorials, and toolsdocumentation, examples, tutorials, and tools
Simple time series identifier, in addition to the DataSimple time series identifier, in addition to the Data
Structure Definition - eg local identifier as part ofStructure Definition - eg local identifier as part of
exchange - the key back to internal data holdings;exchange - the key back to internal data holdings;
existing international 'code' for each series used inexisting international 'code' for each series used in
current exchanges - doesn't vary from country tocurrent exchanges - doesn't vary from country to
country.country.
23. ConclusionConclusion
"Towards Implementation of SDMX - Jan 9-11, 2007"
Move to SDMX V2, recognising need to keep back-Move to SDMX V2, recognising need to keep back-
end systems independent of version changes.end systems independent of version changes.
Importance of establishing business benefit ofImportance of establishing business benefit of
SDMX - it is not just a technology.SDMX - it is not just a technology.
Ideal of a common framework that would guideIdeal of a common framework that would guide
each statistical domain to determine their exchangeeach statistical domain to determine their exchange
requirements and definitions.requirements and definitions.
Need for good registration processes around allNeed for good registration processes around all
aspects of SDMX, and involvement of NSO's inaspects of SDMX, and involvement of NSO's in
governance.governance.
SITC - Standard International Trade Classification ICD - International Classification of Diseases and Related Health Problems
Reason that a new data load was required is that the Data Structure Definition (DSD) is different in some ways to the way the ABS otherwise disseminates NAB data currently. The ideal for the future will be that the data is loaded only once and downstream transformation processes "shape" it to meet the DSD requirement and separately to meet other requirements.