Connecting to the StatXplore API in PowerBI

Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Connecting to the StatXplore API using Power BI
Introduction
The StatXplore API allows users to retrieve data as it is updated. The response sent by the API
can appear challenging to process, but data can be successfully extracted given the consistent
layout of the response.
The response layout requires two queries; one to retrieve labels for each of the fields requested
(such as date or geography), plus an additional query for the data itself. These queries must be
linked to use the labels alongside the data.
This guide is split into two main sections: how to query the API and how to process the
response. The Appendix contains examples of query bodies, a query template, and a full query
example (excluding API Key).
The method described in this guide was initially created with assistance from colleagues at IJYI.

Querying the StatXplore API
Prerequisites
To query the StatXplore API, you will need:
a. A personal API Key. This is obtained by registering on the StatXplore website. Always
use your personal API Key. If the user assigned to your API Key deregisters from
StatXplore, your queries will not function. Once registered, your API Key can be found
in the Account section of the website. Use the link in the top right once registered and
logged in:
b. The body of the query to send to the API. The easiest way to construct this is to use the
StatXplore website to build the table of data you need, then export it as an Open Data
API query (.json). Save this file so that you can refer to it later. This export will need
further editing before use in a query, see Preparing the Query Body.
c. A program to prepare the query body. Notepad++ is recommended, although
alternative software can be used.
d. A template to put your query body into. An example template can be found in the
Appendix of this document.

Preparing the Query Body
The .json file that is exported from StatXplore (see Prerequisites) must be edited before use.
This needs to be done for each query that you use. See the finished Example query body (1) and
Example query body (2) in the Appendix.
1. Open the .json file exported from StatXplore in step B of the Prerequisites in Notepad++.
2. Remove the text relating to date fields from the recodes section, but not from the
dimensions section. This is done so that values from all dates are returned, not only
those dates specified on the StatXplore website. Without this step, queries will not
return new data. Remove the text from and including the comma preceding the date
field name, up to and including the right parenthesis after the ‘total’ line of the date
field (see examples below where the highlighted text should be deleted).
3. Remove the text relating to any other fields for which you want to return all available
data rather than only the values initially specified. Use the same technique as above:
remove the text from and including the comma preceding the field name, up to and
including the right parenthesis after the ‘total’ line of the field. This could be used, for
example, to see values for all the employment indicators for Alternative Claimant Count.

Remove the text relating to these fields from the recodes section, but not the
dimensions section. Removing text from the dimensions section would remove this field
from the query entirely.
4. Replace every instance of a speech mark (") with a double speech mark (""). Note that
this must be the two vertical lines; using the quotation marks in Word (“ and ”) will not
be recognised and will cause the query to fail.

Creating a Query
1. In Power BI desktop, add a new data source and use the Blank Query option.
2. With this new blank query created, open the Advanced Editor. Select everything in the
box, delete it, then paste in the Query Template.
3. The query template creates the variables url, body and header:
• url is the web address used for any query that aims to bring back data from the
StatXplore API. This does not need to change.
• body is the part of the query that tells the API which data to return. Paste your
prepared query body between the speech marks in the body line.
• header is where your personal API Key is added to authenticate access to the API.
Paste your API key between the speech marks in the header line.
The variable response uses the url, body and header variables as inputs to the Web.Contents
function. response is returned as json using the Json.Document function.
Click Done and wait for the response from the API. If your query has worked, you should see
this:

If so, you can now access and transform the data.
If asked for credentials for this data source, select Anonymous credentials and Public privacy
level.

Accessing and Transforming Data
Overview
The response from the API should look like this:
The data returned by the query comes from the fields and cubes rows of this table. fields
contains the labels for your dataset such as geography, date and sex. cubes contains the values
of your dataset. The yellow words List and Record act as links and can be used to navigate
through the json document.
By default, there are no labels assigned to the values in the cubes section. The data is
presented in a hierarchy, but there are no labels assigned to each level in the hierarchy.
Fortunately, the response sent by the API will always follow a set of rules. The rules of the API
response are:
a. The hierarchy of data within the cubes is always in the same order as the List of fields.
b. The labels for each field are always in the same order as the List of records within a
field.
These rules mean that, if expanded correctly, a flat table can be created with each combination
of labels in the order that they occur in the data. The general process for this method is:
1. Transform the list of fields into a single row.
2. Expand each column, working from left to right, to obtain the labels. Each expansion will
create new rows for each combination of existing labels and the newly expanded labels.
3. In a duplicate query, expand the list of values to new rows. Continue expanding to new
rows until the data appears.
4. The order of rows in the labels and values tables is identical. Assign each row in both
tables an index value to allow the tables to be merged or linked.

5. Merge the values table into the label table using the index columns and expand out the
values column into the labels table. Alternatively, create a relationship between the
index columns in the modelling view and use both tables.
Worked Example: Households on Universal Credit
To show more effectively how to transform the data, I have included a worked example of a
connection to the database for Households on Universal Credit. This uses the example query in
the Appendix. The step numbers correspond to the numbers from the Overview.
Step 1: Transform the list of fields into a single row
Begin from the response provided by the API (see Creating a Query). Click on the List in the
fields row.
From the Transform menu, select Convert To Table. This allows the Records to be manipulated.
Expand the column of Records to get a List for the labels for each field (there are two in this
example, area and date). Click the Expand button at the top right of the column. Select only the
items checkbox, then click OK.

The Lists need to be switched from rows to columns. Use the Transpose option from the
Transform menu.
There is now a column for each field in the query.

Step 2: Expand the labels table to obtain labels for each row
Each column should now be expanded to obtain the labels for each field. The rows will be
created in the order that they will appear in the data. As each column is expanded, new rows
are added for each existing row.
Click the Expand button at the top-right of the first column. Select Expand to New Rows. This
will create a new row for each of the labels in this field.
Then click the Expand button again and select only the Labels checkbox.
Then click the Expand button for the third time and select Expand to New Rows.

This has added to labels for the first field. Now, return to the start of this step and repeat the
process for all the field columns, working from left to right, until the rightmost column has
labels rather than Lists. When expanding each column to new rows for the first time, a new row
will be created for each label within the new row, for each existing row. In this example, the
expansion of the second column results in 76 rows for each of the existing rows (Babergh, East
Suffolk et cetera) because the second field (month) has 76 labels.
Once all the columns have been expanded and the labels are shown, the labels table expansion
is complete. Each row will be a unique combination of labels which will appear in the same
order as the values table. Rename the columns and change their types accordingly.

Step 3: Expand the values table
Duplicate the previous query and delete all steps up to and including the ‘fields’ step.
Navigate to the Record in the Cubes row. Then select the Record within that, then the List
within that. This should display a List of Lists.

Next, convert this List to a table from the List Tools menu; click OK in the dialog box that
appears.
The table should now display one column with a List in each row. Each List must be expanded to
get the data values. Click the expand button at the top-right of the column and select Expand to
new rows. Repeat this until the data is displayed.
The data is now displayed with a row for each value. These rows correspond exactly to the
order of the rows in the labels table.
Rename the values column and change its type accordingly.

Step 4: Create index columns
The labels and values tables are now complete and ready to be linked. For both queries, add an
index column using the button in the Add Column tab. Use the same settings for the index
columns for both queries.
The tables can now be linked using modelling (relationships) or transformations (merging
queries).

Step 5a: Linking by merging tables
To use the merging queries approach, open Transform data and select the labels table. From
the Combine section of the Home tab, select Merge Queries.
Select the index column from the labels table. Then select the values table from the first drop-
down box and select the index column. A Left Outer join will be required as the aim is to add
the value column from the values table to the labels table. All rows should match, if not there
has been an error in the expansion of the tables or the indexes have been created using
different settings. Click OK.

Click the expand button in the top-right of the new column. Expand out only the values column.
This will add the values to the labels table, resulting in one table that can be used to filter and
display data as required.

Step 5b: Linking by relationships
To use the modelling approach, load the data and switch to the Model view. Click and drag the
tables you will be using so that they are close together.
Click and drag the index column from one table to the other. This will automatically create a
relationship between these columns.
The relationship should always be of the type 1:1. This will be automatically calculated. If the
relationship is not of this type, there has been an error in the expansion of the tables or the
indexes have been created using different settings. To make the report easier to use, hide the
index columns from the report view using the eye icon.
The fields from the labels table can now be used to filter and display the data in the values
table.

Appendix
Example query body (1)
{
""database"" : ""str:database:UC_Households"",
""measures"" : [ ""str:count:UC_Households:V_F_UC_HOUSEHOLDS"" ],
""recodes"" : {
""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" : {
""map"" : [ [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000200"" ], [
EGION:E07000244"" ], [
EGION:E07000202"" ], [
EGION:E07000203"" ], [
EGION:E07000245"" ] ],
""total"" : true
}
},
""dimensions"" : [ [ ""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" ], [
""str:field:UC_Households:F_UC_DATE:DATE_NAME"" ] ]
}

Example query body (2)
{
""database"" : ""str:database:ACC"",
""measures"" : [ ""str:count:ACC:V_F_ACC"" ],
""recodes"" : {
""str:field:ACC:V_F_ACC:UK_COA"" : {
""map"" : [ [
""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000200"" ], [
""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000245"" ] ],
""total"" : true
}
},
""dimensions"" : [ [ ""str:field:ACC:V_F_ACC:UK_COA"" ], [
""str:field:ACC:F_ACC_DATE_new:DATE_NAME"" ], [ ""str:field:ACC:V_F_ACC:EMP"" ] ]
}

Full example query
For privacy reasons the API key has been omitted. Copy and paste your personal API key
between the speech marks after APIKey= to get this query to work.
let
url = "https://stat-xplore.dwp.gov.uk/webapi/rest/v1/table",
body = "{
""database"" : ""str:database:UC_Households"",
""measures"" : [ ""str:count:UC_Households:V_F_UC_HOUSEHOLDS"" ],
""recodes"" : {
""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" : {
""map"" : [ [
EGION:E07000200"" ], [
EGION:E07000244"" ], [
EGION:E07000202"" ], [
EGION:E07000203"" ], [
EGION:E07000245"" ] ],
""total"" : true
}
},
""dimensions"" : [ [ ""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" ], [
""str:field:UC_Households:F_UC_DATE:DATE_NAME"" ] ]
}",
header = [ #"APIKey"="" ],
response = Web.Contents(url,[Content=Text.ToBinary(body),Headers=header]),
Source = Json.Document(response,1252)
in

Source

Query template
let
url = "https://stat-xplore.dwp.gov.uk/webapi/rest/v1/table",
body = "",
header = [ #"APIKey"="" ],
response = Web.Contents(url,[Content=Text.ToBinary(body),Headers=header]),
Source = Json.Document(response,1252)
in
Source

Connecting to the StatXplore API in PowerBI

Connecting to the StatXplore API in PowerBI

Connecting to the StatXplore API in PowerBI

Recommandé

Contenu connexe

Tendances

Tendances(20)

Similaire à Connecting to the StatXplore API in PowerBI

Similaire à Connecting to the StatXplore API in PowerBI(20)

Plus de Office for National Statistics

Plus de Office for National Statistics(20)

Dernier

Dernier(20)

Connecting to the StatXplore API in PowerBI