ONS business data and statistics user event

ONS business data and
statistics user event
19 April 2023
slido #28361

Welcome
Abbe Williams
Assistant Deputy Director
Surveys and Economic Indicators
Office for National Statistics
slido #28361

Agenda
10:00am to 10:05am Welcome – Abbe Williams, Assistant Deputy Director, Surveys and
Economic Indicators, Office for National Statistics
10:05am to 10:15am Opening address – Mike Keoghan, deputy National Statistician, Office for
National Statistics
10:15am to 10:45am Business survey data, their statistics and uses – Jon Gough, Office for
National Statistics
10:45am to 11:00am Questions and Answers
11:00am to 11:10am Break
11:10am to 11:40am Transformation of business statistics – Craig Taylor, Deputy Director for
Business Statistics Transformation, Office for National Statistics and Beck
Keane, Office for National Statistics
11:40am to 11:55am Questions and Answers
11:55am to 12:05pm Break
slido #28361

Agenda
12:05pm to 12:35pm ONS future plans – Andrew Allen, Office for National Statistics and Tony
Chapple, Office for National Statistics
12:35pm to 12:50pm Questions and Answers
12:50pm to 1:20pm Economic microdata transformation – Rabiya Nasir, Russell Black and
Yohannes Lemma, Office for National Statistics
1:20pm to 1:35pm Questions and Answers
1:35pm to 1:40pm Closing remarks – Darren Morgan, Director, Economic Statistics Production
& Analysis Office for National Statistics
slido #28361

Questions can be submitted via slido.com using code #28361.
You can also access slido via the link in the chat box.
slido #28361

Opening address
Mike Keoghan
Deputy National Statistician and Director General,
Economic, Social and Environmental Statistics Group
slido #28361

Business Survey data,
their Statistics and Uses
Jon Gough
slido #28361

Business Surveys
• Numerous surveys which help to measure the economy,
labour market and socially
• Business surveys cover all sized businesses covering a
range of topics and frequencies
• Financial Variables
• Structural and short-term surveys
• Feed into and meet user requirements

Retail Sales Index
• Retail sales data are collected from a sample
of approx. 5,000 retailers across GB
• The primary purpose of the Retail Sales Index
(RSI) is to produce a short-term measure of
the changes in the volume and value of sales
of goods by retail businesses
• The RSI feeds into estimates of Gross
Domestic Product (GDP)
• The value and volume measures of retail
sales estimates are used by the private and
public sector particularly by the Bank of
England (BoE) and HM Treasury to assist in
informed decision-making and policymaking

Monthly Business Survey
• Monthly Business Survey (MBS) is a monthly
sample of around 6,000 businesses
• Feeds both the Index of Production (IoP) and
Index of Services (IoS) which in turn is used to
create monthly and quarterly GDP
• Two of the primary data sources used in
the compilation of monthly and quarterly GDP
estimates; IoP and IoS account for approx.
93% of the economy
• Used extensively by Government stakeholders
to inform economic policy and future forecasts
• Used by BoE at Monetary Policy Committee
meeting to help inform interest rate decisions.

Construction
• Surveys approx. 8k businesses per
month
• Publish by type of work – New and
Repair & Maintenance
• Nine sectors incl. housing, infrastructure
and non-housing
• Sub-sector and sub national estimates
• Component of MGDP and QGDP
• Used by other government departments
such as BoE, HMT, BEIS and DLUHC

Business Insights and
Conditions Survey
(BICS)

What is BICS
• The ONS created a new innovative
fortnightly business survey, initially to
monitor the impact of COVID-19,
now geared towards assessing
business sentiment
• The BICS voluntary fortnightly
business survey has provided us
with a rapid and flexible survey to
respond quickly to changing policy
needs
• Detailed results are published each
fortnight with headline figures
published each week in a Faster
Indicators (now called RTI) bulletin
Financial
performance
Workforce
size
Furlough
Grants and
Schemes
EU Exit
International trade
& supply chains
Net zero
Shortage of workers
Global supply
chains
Increase in prices
(energy)

Business Insights and Conditions Survey
• Fortnightly survey
• Sample of around 39,000 businesses
• Response rate 23-25%
• ONS first ‘pulse’ business survey
• Collects qualitative information
• Initially collected the extent of trade during the pandemic and normal expectations for that
time of year but expanded to cover range of topics:
• Impact of EU Exit on trade
• Environment and supply chains
• Labour shortages and recruitment challenges
• Government schemes to assist businesses during lockdowns
• Price increases, including the energy prices

Capturing data from businesses to meet users
diverse needs
• Flexibility of the online survey and BICS processing
systems allow us to add new questions quickly
• The question bank of all the BICS questions for each wave
is publicly available
• We work closely with stakeholders on their priorities,
rotations for the questions and this allows us to keep up
to date with the latest emerging issues and gaps in
data
• With only a two-week window each wave we:
• Agree content and design questions with the stakeholder
• Work closely with ONS methodology teams to further refine questions
• Test the survey before it goes live
Agree Content
Design
Questionnaire
Test

No longer a “COVID” survey, from April 2022,
BICS moved towards business tendency survey
with predictive power
Regular
question and
comms
reviews
Introduction of
core
questions
Adapting to
free text
responses
provided
businesses
• Taken steps to transform BICS into a business tendency
survey
• Leading indicator on the real economy with potential for
monitoring and forecasting business cycles
• Collect consistent responses monthly over time (instead of
'normal times')
• Indicators that have predictive power (output, prices
employment)
• Valuable for policy makers and forecasters
• Focus on 'core' and 'non-core' themes fortnightly

Taken steps to transform BICS into a business tendency survey
Core – leading indicators with predictive
power
Non-Core (Adhoc) – broad themes to help
answer policy questions
Exporting and
Importing
•Compared with a year ago
•Increase in challenges/ disruption
•Support received
Business
Concerns
•Falling demand
•Energy prices
•Inflation
Global Supply
Chain
•Disruption
Workforce
•Location changes
•Worker shortage
•Wages / redundancies
Environment
and climate
•Concerns
•Action taken
Turnover
•Turnover compared with previous month
•Challenges impacting turnover
•Turnover expectations
Prices
•bought/ sold compared with previous month
•Price pass through to customers
•Energy
Stock and
Capital
expenditure
•Stock levels
•Expectations for capital expenditure
Workforce
•Number of employees compared with previous month
•Expectations for staffing costs
Business
Confidence
•Overall performance
•Potential of insolvency
•Cash flow

Use of BICS: Monitoring Cost of Living

Use of BICS: BICS vs monthly GDP*
Source: Office for National Statistics – Business Impact of Coronavirus (COVID-19) Survey
Trends from businesses in
BICS are also seen to be
broadly reflect the published
UK monthly GDP estimates,
where BICS is published much
earlier than the official monthly
GDP estimate.
Trend follows more closely
since the change to BICS in
April 2022

Use of BICS
Recent trends in UK food and drink
producer and consumer prices:
January 2023
Nature and biodiversity risks
monitored by 3% of UK businesses
ahead of COP15
Rising business insolvencies and high
energy prices
Food and drink service firms are
most likely to cut trading to tackle
energy costs

Future developments
• Reviewing content – questions and responses
• Identifying where BICS can support with data gaps
• Improving response rates where possible
• Subnational estimates weighting methods
• Exploring further predictive power of BICS to support Retail
Sales Index, Construction

Structural Business
Surveys
19 April 2023

Annual Business Survey and Annual Purchases
Survey
• Annual Business Survey (ABS) and Annual Purchases Survey (APS) form essential
building blocks in the supply-use framework
• Estimates from ABS make up three quarters of output by industry, and provide the majority of data
on intermediate consumption
• Provides estimates of Turnover, Purchases and aGVA at detailed industry level
• In 2020, approximate gross value added at basic prices (aGVA) of the UK non-financial business
economy was estimated to be £1,194.9 billion
• APS gives the detail of which products are being consumed
• They directly impact other areas of National Accounts including:
• Gross operating surplus
• Gross fixed capital formation
• Taxes and subsidies
• ABS & APS help inform policy decisions
• Supplying data to other government departments for specific analysis e.g., identifying energy
intensive industries

UK Manufacturers sales by product - PRODCOM
• Products of the European Community (PRODCOM) is a long standing, large ONS
business survey and is the primary source for measuring the UK manufacturing
industry broken down by the products the industry produces
• The total value of UK manufacturers’ product sales was £400.8 billion in 2021
• Purpose: Main uses of PRODCOM outputs are to inform the UK supply and use
tables, it is also used to develop a sampling frame for the Producer Price Index
(PPI) which is used to monitor inflation in sales at the “factory gate”
• Key users: ONS (National accounts and Prices) Devolved Administrations, DIT,
BEIS, MOD
• Policy Impact: PRODCOM tracks manufacturing in the UK at product level and is a
key source used when considering policy setting for the production industry
• Supply chain issues being experienced in the UK, PRODCOM data has been used to
help identify vulnerable products and when merged with other data, vulnerable countries

Annual Survey of Goods and Services (ASGS)
• Relatively new ONS survey, considered as an experimental output, covers the UK
services industry collecting annual turnover values broken down by the
goods/services provided
• Purpose: Main uses for ASGS outputs are to inform the UK supply and use tables, it
is also used to develop a sampling frame for the Services Producer Price Index
(PPI)
• Key Users: ONS (National accounts and Prices), Devolved
Administrations, DIT, BEIS
• Policy Impact: As an experimental output, ASGS is not directly used to inform policy,
more indirectly in that ASGS is used to inform wider ONS outputs that are used to
inform and base policy making decisions on for example GDP and SPP outputs.

Q&A
Questions can be submitted via slido.com using code #28361.
You can also access slido via the link in the chat box.
slido #28361

Transformation of
business statistics
Craig Taylor
Deputy Director for Business Statistics Transformation
Beck Keane
slido #28361

What we will cover…
• Update on transformation of Research and Development
statistics progress
• Update on plans for some of development of some of our
larger structural statistics
• High level summary of different phases of business statistics
transformation

Transformation of Research
and Development statistics
Update on progress
Craig Taylor
Deputy director, Business Statistics Transformation
slido #28361

Summary of the work
Mission
Vision
Drivers
Objectives
Review & Development of R&D Statistics Completed by 2024
By 2024, UK R&D statistics are internationally recognised and trusted as high quality and responsive to user
needs
Differences
between
ONS/HMRC
estimates
New data
sources
Levelling
up
Need to
conduct a
review of
R&D statistics
Understand and explain
differences in different
data sources
Use of available
administrative data
sources
Provide more granular
level statistics
Be flexible and adaptive
to user needs
Expected
outcomes /
benefits
Better informed decisions and
improved policy evaluation related
to R&D expenditure in the UK
Increased confidence in R&D data
Increased responsiveness and
flexibility

R&D transformation timeline
End2022 May2023
July2023
April2023
March
2024
Spring2024
Improved Research &
Development statistics
published (R&D) – phase 1
Transformed Business Enterprise
Research & Development (BERD)
survey despatched on EQ
GovERD survey
despatched with
additional question/s and
improved guidance
Experimental Regional
Government Expenditure on
R&D Estimates published
Transformed Research &
Development statistics
(phase 2) published (note
aim for BERD to be
published)
Complete
Further developed BERD survey
(possibly including new questions)
despatched

Reminder of R&D phase 1
Business Enterprise R&D
• Identified BERD methodological improvement and
incorporated into BERD (2014-2021) and GERD
releases (2018-2020)
• Reduced BERD release due to breakdown
uncertainty (UK figure National Statistic;
breakdowns NS status paused) and limited
microdata available
• Article explaining uplift approach and validation
Higher education R&D
• Secured use of TRAC data as alternative source
 Improved HE R&D statistics including HE own funded and
performed R&D (£4.9bn in 2020) and wider indirect costs of
R&D and incorporated into GERD statistics in November
2022
• Changes in total HE R&D recorded as part of GERD statistics
and taken back to 2018
TRAC data
• Identified best practice on R&D collection and
produced recommendations for depts
• Developed short-term baseline for mission based
around 3 large R&D funding departments
 Baseline estimates published on 17 April 2023.
Government R&D

Regional (ITL1) Public Funded R&D Expenditure
Baseline: Background
The Levelling Up White Paper R&D mission states:
“By 2030, domestic public investment in R&D outside the Greater South East will increase by at least
40%, and over the Spending Review period by at least one third. This additional government funding
will seek to leverage at least twice as much private sector investment over the long term to stimulate
innovation and productivity growth.”
In terms of Government expenditure on R&D, this implies the following data requirements:
Short-term development of baseline data for
Government spending on R&D at regional
level.
Improvement to the GOVerd survey to collect
more accurate regional data on Government
R&D expenditure (2024).

Sample design Survey design
Take-on and
validation
Results
processing
Dissemination
BERD: Sample design which
addresses under coverage
- Increase of sample size to 40k
(from 5k)
- Samples directly from business
register
- Improved reference list using
HMRC tax data to target the largest
business with R&D expenditure
BERD: Sample design which
optimises at sub-national level for as
much as sample size allows
- Increased use of long form to capture
more detailed information
BERD: Improved guidance for
existing questions on survey form/s
BERD: Move surveys to EQ
BERD: Improved coverage of sub-
national R&D information
BERD: Improved coverage of R&D
product information
BERD: New take-on and
validation system which has
enabled streamlining of validation
rules and move to strategic
platform
BERD: New methods for
processing results e.g.,
estimation, imputation
BERD: New results system which
enables incorporation of new
methods, delivers improvements
to quality assurance approach
BERD: Improvement to sub-
national data through data
collection and learning from
baselining methods
All: R&D outputs provide inputs into
the National Accounts – part of core
delivery
BERD: Back series (required to
support R&D publication and
delivery into National Accounts)
GovERD: New questions in
response to stakeholder demand
GovERD: changes to absorb
question changes within data
collection
GovERD: changes to absorb
question changes within results
processing
High level scope of R&D phase 2

Ongoing research areas
While focus remains on delivering our challenging work for phase 2 work (predominantly transitioning our end to end system
design onto new platforms to support the new sample design), there are some specific work that continues alongside, to help
shape and inform future workstreams:
• R&D international comparisons- carrying out some research into international approaches to measuring R&D, to understand what future
improvements can be made
• R&D back series development
• R&D and the national accounts- understanding the current inclusion of R&D activity in the national accounts, how this is captured and how the
new R&D work can be implemented
• R&D additional analytical work- with the increase in sample size due to the new design, we are committed to increasing our analytical resource
to make the best use of the existing sample size. It’s likely this work will be used for internal purposes, including refining future surveys and
methods, as well as helping make sure the quality of our statistics can be maintained
• GOVERD- learning from the one-off baseline data collection exercise to make improvements to our regular GOVERD publication
• GOVERD- In conjunction with GO-Science, developing questions to be added to GOVERD to collect more granular information on purchased R&D
by region, expenditure on Official Development Assistance (ODA) and technology areas i.e Artificial Intelligence
• GOVERD- taking part in a pilot scheme in collaboration with GO-Science to help better prepare departments to complete GOVERD returns

User engagement
Engagement so far Next Steps
Huge amount of interest and user engagement, which has
included:
 User consultation
 Presentations and engagement at vast array of events,
conferences e.g. Better Statistics Conference
 ONS Seminar on 6th December on measurement of R&D
(BEIS; Go Science; OECD; British Academy presented)
Engagement with stakeholders has been wide ranging and
has included:
 Stakeholder events such as Business Statistics User event
 Continue to engage with stakeholders through established
forums highlighted
 Article outlining development progress (summer 2023)
 Additional article including initial indications of impact to
GDP etc (Autumn 2023)
 ONS seminar focused on R&D (end 2023)
Devolved Administrations British Academy, Universities UK
HMRC, BEIS, HMT, DLUHC R&D Strategy Board
Go Science Council Science and Technology (CST);
Chief Scientific Advisors
OSR Various academics inc LSE, ICL
External R&D expert group inc OECD and Nesta BofE

Annual purchases survey (APS)
Annual Business Survey (ABS)
Update on plans
Beck Keane
Assistant deputy director, Business Statistics
Transformation
slido #28361

Aim
Drivers
Opportunities
To review and develop the Annual Business Survey (ABS) and Annual Purchases Survey (APS)
for an efficient, flexible and future proof solution
Improve responder experience and streamline data collection
Respond to emerging needs (including improved granularity)
Produce national statistics as endorsed by OSR
Strategic systems and efficient processes
Improved methods and improved quality assurance
New data sources
Benefits
Increased responsiveness and flexibility, reduce risk of errors
Better informed decisions
Improved policy evaluation
Annual Business Survey (ABS)
Mission Statement

APS and ABS progress to date
• Understanding areas for redevelopment across the
design
• Using new technologies to move data collection online
(2024 collection)
• Reviewing end to end methods
• Gathering and evaluating our user requirements
following our consultation

APS where next…
• Improving flexibility by rationalising questionnaires
• Improving respondent experience through questionnaire
design and automated validation
• Configuration changes to processing systems to improve
business processes for APS production

ABS where next…
• Exploring strategic system to better support flexible
response to user needs
• Developing methods to support user requirements
• Improving respondent experience through questionnaire
design and automated validation

User engagement
Engagement so far Next Steps
Engagement activities have included:
• OSR published ABS review
• ONS consultation on ABS (June 2022)
• Government user groups (next meeting 25th April
2023)
Engagement has focussed on known users of ABS
statistics including
• ONS response to ABS consultation (May 2023)
• ONS response to OSR report on ABS review
(Summer 2023)
• Sharing plans for next steps (summer 2023)
• Identify additional users of ABS and APS
Are you a user of our ABS and APS data? We would love to hear from you to make sure we
communicate plans and understand requirements of our data abaps@ons.gov.uk
Devolved Administrations Department for Trade
BEIS, HMRC, HMT OSR
DCMS Government User Group (GuG)

Business statistics
transformation phases of
delivery
Craig Taylor
Deputy director, Business Statistics Transformation
slido #28361

R&D Phase 1
• Address under-coverage in BERD
sample design by implementing an
uplift to address known bias outside of
core processes
• Sub-national baseline work in response
to Levelling Up White Paper
1
R&D Phase 2
• New sample design to address under-
coverage & optimisation for agreed
breakdowns
• Move to electronic data collection for all
R&D surveys, inc. improved guidance
• Improved coverage of collecting
information on sub-national & type of
R&D conducted
2 Structural Surveys
• Move of APS to electronic data
collection
• Revised ABS questionnaire as part of
move to full electronic data collection
• Full end to end transformation of ABS
data take-on & results processes
systems
3
Further phases of
transformation
4
BST PROJECT – PHASED DELIVERY
• New estimation & imputation methods
• New results systems enabling
incorporation of new methods &
improved QA processes
• Outputs able to be incorporated into
wider National Accounts

ONS future plans
Andrew Allen
Tony Chapple
slido #28361

Statistical Business
Register (SBR)
Andrew Allen
slido #28361

52
Background
• Inter-Departmental Business Register (IDBR) was
introduced in 1994. Improved constantly since, but
technology limits further development.
• IDBR provides sample frame for all business surveys
and is used for analysis of UK business population.
• Will be replaced by :
• Statistical Business Register launch 2024

Statistical Business Register (SBR)
Transformed register of UK businesses, covering the functionality of the
IDBR
Core functions:
• Underpins Economic statistics
outputs
• Create statistical units
• Create and manage frames
• Create and manage samples
• User interface and analytical
extracts
Benefits:
• Modern technology.
• New data sources – full companies house
population, corporation tax and self assessment
– will lead to full business population (additional
+2m businesses).
• More variables, improved sampling & analytics
• Improved quality and granularity.
• Greater use across Government.

Phase 0
(Discovery)
Phase 1
(Research)
Phase 2
(Alpha)
Phase 3
(Beta 1)
Phase 3 cont.
(Beta 2 and 3)
Phase 4
Migration
SBR Discovery:
An internal ONS
Discovery Phase has
taken place.
Research Phase:
Carry out a research
phase to understand
the needs of the users
and system
requirements and
formulate solution
options.
Alpha Development:
Create and
demonstrate a
prototype to verify that
the solution is
achievable and meets
user needs.
Prioritising Beta
requirements and
engaging with
stakeholders to
validate.
Beta Development:
Incrementally develop
and regularly
demonstrate a
production-ready
solution.
Mostly technical
environment set -up
Beta Development &
transition to DST:
Incremental
development and roll-
out of new features.
CGI will provide
technical and project
support, particularly at
key points when rolling
out the system. This
will include during
operational tests, when
the system is launched
to staff and managers
and when the system is
first used by users.
Migration of Surveys
and change
management:
1. Enhanced support
2. Support business
surveys to transition to
the use of SBR by
creating similar
population as IDBR.
3. Expand the sample
population for business
surveys
SBR Phases
COMPLETE COMPLETE COMPLETE 30 Mar 23 B2: 16 Jan 24
B3: 29 Mar 24
30 Jun 24

55
IDBR data sources
• We aim to capture economic activity and not just
business registration
• IDBR is based on VAT and/or PAYE returns: company
registration data used to assist matching.
• Small companies without a VAT/PAYE are not held.
• Hence IDBR has 2.0m Companies. Companies House
has 4.6m effective registrations.

56
SBR new data sources
• HMRC Corporation tax dataset
• will provide better financial variables
• will facilitate the use of the full companies house
population
• HMRC Self Assessment
• will allow better coverage of very small businesses.
Giving us the full business population for the first time.

57
Phased data source improvements
• The SBR technical build will be complete in 2024 - which will
allow us to flexibly add data sources.
• New data sources will be added as they become available.
• For example, ONS is still negotiating the Self-Assessment
supply. Rules and processes will be needed to then adopt on the
SBR.

58
SBR benefits will feed into downstream products
• SBR will improve Business Structures Database – extra
business population and better time-stamping of
variables etc.
• Register based publications will transition to full
business population e.g. UK Business and Business
Demography
• Timetable and details will be agreed later in the project.

59
Business survey outputs
• We will need to assess and work out how and when to
use new register coverage.
• The aim is to be able to replicate the IDBR population
to enable continuity, while the impact of extra
businesses is considered.

In partnership with
Secure Research
Service (SRS)Transition
Tony Chapple – Head of SRS Transition
tony.chapple@ons.gov.uk

Potted history VML <> SRS <> IDS
VML
Standalone
/ VML
Tactical
VML Private
Cloud-based
(PSN)
SRS Private
Cloud-based
(Internet)
SRS
Technology
Refresh
15+ years providing secure
Research Access to deidentified
unpublished data using 5-Safes
Framework

VML
Standalone
/ VML
Tactical
VML Private
Cloud-based
(PSN)
SRS Private
Cloud-based
(Internet)
SRS
Technology
Refresh
Framework
Advances in technology have opened
the door for developments in the way
users access data. IDS will build on
the foundations provided by the SRS,
and will, in time, replace the current
service.

VML
Standalone
/ VML
Tactical
VML Private
Cloud-based
(PSN)
SRS Private
Cloud-based
(Internet)
SRS
Technology
Refresh
SRS Transition
to IDS
Framework
IDS will represent a change from the way researchers access data in the SRS.
Supporting users in making the change, a dedicated project team is in place
to plan the transition and ensure that users move from the SRS to IDS as
smoothly as possible.

Capabilities
Capability Roadmap
(Process & Technology)
Clear
outputs
Supplementary
Ingest
Capabilities
(Process & Tech)
Tooling Roadmap Tools
Data Roadmap Data
Tools
STATA
SPSS
R,Python
Data
Census
LFS
ASHE
Key: SRS transition needs Wider IDS needs
Context setting – Strategic approach to transition
Integrated
Data
Service
(IDS)
Secure
Research
Service
(SRS)
Analysis
IDS Strategic Roadmaps
User Experience

Data
Tools
Capabilities
(Process & Tech)
SRS
Transition
Funnel Capabilities
(Process & Tech)
Tools
Data
Analysis
Availability
Tranche
planning
Context setting – Strategic approach to transition

SRS-IDS Transition: High Level Approach
Initial Discovery Mobilisation
Initial Proof of Concept
Analysis & Planning
Tranches
Final Decommissioning &
Handover
Plan
Migrate
Accept
Test
OUTPUTS:
• “As Is”
• “To Be” Gap Analysis
• High Level
Requirements
• Transition Strategy
OUTPUTS:
• Transition Kick Off
• Identify “Air Traffic
Controllers”
• Detailed Planning &
Requirements
• Data Owner Engagement
• Comms Kick Off
• Plan E2E Pilot OUTPUTS:
• Transitioned Data
• Transition Project
• Transition Users
• Lesson Learnt
OUTPUTS:
•SRS Staff Transitioned
•SRS Decommissioned
•IDS Service Level
Acceptance
L0 SRS Decommission
OUTPUTS:
• Detailed Transition
Plan
• Lessons Learnt
IDS Tooling Roadmap
IDS Data Roadmap
SRS-IDS Staff Transition & Service Development Demand Framework
DISCOVERY / DESIGN & DEVELOPMENT TRANSITION TO SERVICE SRS CLOSURE
IDS Capability (Process & Technology) Roadmap

Transition Complexities……
Staff
Projects
& Users
Data
Org
Connectivity
(Cloudflare)
Tooling
Projects
& Users
Data Owner
Agreements
DEA
Data

High Level Project Transition
1.
Engagement
Willingness
to transition
?
Finish in
SRS
2.
IDS Onboarding
Project /
Account
Setup
Original
Data made
available
Training & Guidance
3.
Dual Running
SRS
IDS
3A.
Code extraction
Output
from SRS
Ingest into
IDS
4.
Accept into IDS
Shutdown
Project in
SRS
Continue in
IDS
69

Some of the Differences …
Connectivity & Access
Cloudflare
User Experience
Cloud native <> Virtual Desktop Infrastructure
Dataviews ‘Big Query/ SQL’ <> ‘flat files etc.
Research
Tooling
Code & Package use

Summary
This is the start of the Transition Journey
• We will be providing regular communication updates to all users
A commitment to minimise disruption as much as possible
• Drive forward PoC (both data and project transition)
• Front load the transition of Data, transition Projects when appropriate
• Transition staff to IDS, according to ramp down / ramp up of SRS / IDS
• Decommission SRS
Next Steps

In partnership with
Thank you !
Tony Chapple - tony.chapple@ons.gov.uk

Economic Microdata
Transformation
Rabiya Nasir, Russell Black and
Yohannes Lemma
slido #28361

Content
• Role of the Economic Microdata Transformation team
• Longitudinal Business Database
• Annual Respondents Database X and Capital Stocks
• HMRC Trade in Goods data in SRS
• Microdata plans for 2023/24

• We are experts on transforming and analysing firm-level data.
• Our research links and develops microdata sources to improve
our understanding of the economy and how to measure it.
• We make firm-level data more accessible to support external
research in government and academia.
• Work closely with Economic Statistics Centre of Excellence
(ESCoE)
Who are Economic Microdata Transformation

Motivating topics
• Productivity: distribution and drivers at firm level
• Business dynamism, competition and markups
• Management and organisational structure
• Role of businesses in Net Zero and ‘levelling up’
• Better and faster GDP estimates

Longitudinal Business
Database (LBD)
Economic Microdata Transformation
LBD Team

The Longitudinal Business Database (LBD)
• New set of firm-level spines of business references
• It differs from existing Inter-Departmental Business Register (IDBR) linked dataset:
 Quarterly cycle frequency
 Uses information from two adjacent snapshots to establish longitudinality
 Pertinent rules to capture and define economic criteria
• First prototype to be made available on the Secure Research Service (SRS)

Benefits of the LBD project
• Quarterly frequency
 Provides researchers with tools to address wide ranging research
questions
• Enable multi-level linking of survey, administrative and novel data
 A single entry of a firm in a given quarter displays the identity-links of
all of its business units registered on the IDBR
• Flexibility to accommodate to any research purpose
• Give end users the capability to produce innovative and insightful research

Variables of interest
• Scope for multi-level analysis  firm-group, firm, establishment-level
unique identifiers
• Activity status of the unit  unit births, deaths, incumbents per quarter
• Turnover/Employment of the unit  productivity, job creation, destruction,
job reallocation etc
• Industrial Classification code of the unit  sectoral/industry analysis
• Region code of the unit  regional analysis

Outputs from the LBD
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1999Q1
2000Q1
2001Q1
2002Q1
2003Q1
2004Q1
2005Q1
2006Q1
2007Q1
2008Q1
2009Q1
2010Q1
2011Q1
2012Q1
2013Q1
2014Q1
2015Q1
2016Q1
2017Q1
2018Q1
2019Q1
2020Q1
2021Q1
2022Q1
Enterprises
(millions)
continuing_active death birth reactivation
0
0.5
1
1.5
2
2.5
3
1999Q1
2000Q1
2001Q1
2002Q1
2003Q1
2004Q1
2005Q1
2006Q1
2007Q1
2008Q1
2009Q1
2010Q1
2011Q1
2012Q1
2013Q1
2014Q1
2015Q1
2016Q1
2017Q1
2018Q1
2019Q1
2020Q1
2021Q1
2022Q1
Percentage
Employment
% of employment from births % of employment from deaths

• Linked dataset - LBD and PAYE to measure employment
dynamics
Main Findings
• Business Dynamism has declined in the UK
• This decline has taken place in all sectors of the economy.
• Older micro firms are exiting the market less than before.
Research using the LBD

Next steps
• Data to be made available on the SRS
• Kick off downstream analyses using the dataset
• Develop and make available the other spines

Annual Respondents Database X (ARDx)
• “Structural Business Survey” has changed several times. Annual
Survey of Manufacturers, Annual Business Inquiry, Annual
Business Survey…
• ARDx: began in the 2000s. Effort to unite this data into one
dataset for researchers, so there is an easy point-of-entry

ARDx – Annual Respondents Database X
• “Structural Business Survey” has changed several times. Annual
Survey of Manufacturers, Annual Business Inquiry, Annual
Business Survey…
• ARDx: began in the 2000s. Effort to unite this data into one
dataset for researchers, so there is an easy point-of-entry
• In practice:
• The content/structure/coding of the surveys is very common, and the
changes in processing are very minor

The original intention was for ABI1 and ABI2 to cover the same companies, to
maximise availability of data when linked.
This link was broken in 2006. While ONS will almost always attempt to be
comprehensive for large companies, there are smaller companies with ABI2 but not
ABI1
Pre 1998 1998-2008 2008/2009 onwards
Employment data: Annual Employment
Survey
Annual Business
Inquiry Part 1
Business Register
Employment Survey
Main economic
performance data:
[several previous
surveys]
Annual Business
Inquiry Part 2
Annual Business
Survey

ARDx 4.0 (April 2023)
• Code completely revised
• Bugs fixed around VAT in the calculation of GVA for
retail, food and accommodation
ARDx 4.1 (Later 2023)
• Fixes for imputations

For researchers
• Management of the SRS still by discrete datasets
• Researchers should request Annual Business Survey
and ARDx datasets for their project

Capital stocks
• Official statistics surveys do not cover company-level capital
stocks (ONS would rather take industry-level investment and use
its own deprecation assumptions to calculate national balance
sheet)
• A key motivation for business survey panels going back to the
past is to estimate company-level capital stocks. Company-level
capital stocks are necessary for production function estimation
and total factor productivity statistics, which are important in the
economics literature.

Capital stock estimates in ARDx dataset
• Will be updated to 2020
• Now at 3 asset level
• Buildings, land and structures
• Software and databases
• Other
• Small updates to imputation

Capital stocks estimates limitations
• Business surveys are not the national accounts. From new years of data it
will be more apparent of where they differ.
• Small and medium companies:
• Surveyed infrequently or only surveyed once. The information content is employment and
industry history from the Inter-Departmental Business Register, and not capital-specific
company data
• Large companies:
• In theory 2/3 of investment is by large companies, covered comprehensively by
ABI2/ABS
• In practice M&A, propcos, this is only the beginning of the problem of using this data in
analysis
• In any case, gap for company-level stocks for the intangible capitals

Datasets in SRS: Linked
HMRC Trade-in-
Goods/IDBR dataset

Inter-Departmental Business Register (IDBR)
ONS-maintained database of companies, from Companies House,
tax data and ONS surveys and reporting

Linked HMRC Trade-in-Goods/IDBR dataset
• HRMC Trade-in-Goods dataset: collected by HMRC for
VAT units at the 9 digit VAT reference, and stored in the
HMRC data lab
1. Dataset linked to the IDBR
2. VAT units do not match cleanly to IDBR enterprises. Trade
values have been apportioned across IDBR enterprises and
reporting units
3. Made available over SRS

Limitations
• Access cannot be given by Assured Organisational
Connectivity. Whole SRS project will be limited to safe
rooms.
• Currently, commodity detail is 4-digit Harmonised
System

Timing
• June 2020: data for 2005-2016
• August 2022: data for 2008-2020, and small change in
method to use more precise weights in apportionment
• Timing for 2021 and 2022 data to be determined

Future plans
• Deliver the next wave of the Management and Expectations Survey.
• Contributions to international microdata projects: CompNet and
OECD.
• Using tax administrative data for firm-level statistics.
• Analysis of business’ energy spending, using the Annual Business
Survey and Annual Purchases Survey.
• Supporting the Integrated Data Service team with migrating
our datasets from SRS to IDS.
• Collaborating with other business data and transformation teams
across ONS to support their work with microdata insights

Closing remarks
Darren Morgan
Director
Economic Statistics Production & Analysis
slido #28361

Thank you for attending our user
event
You can keep up to date on all upcoming events via ons.gov.uk/economicevents
If you would like to ask a question or provide any feedback, please do so via
economic.engagement@ons.gov.uk
slido #28361

ONS business data and statistics user event

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à ONS business data and statistics user event

Similaire à ONS business data and statistics user event (20)

Plus de Office for National Statistics

Plus de Office for National Statistics (20)

Dernier

Dernier (20)

ONS business data and statistics user event