Let's take a look at how to improve the download of reports using the AdWords API.

1. Best practices in
reporting

2. Agenda
0. Why this talk?
1. Best practice #1
2. Best practice #2
…
8. Best practice #8

3. 0. Why this talk?

4. Why this talk?
1. Effective reporting is good for Google
a. Less resources spent to handle requests
b. Less issues to investigate

5. Why this talk?
1. Effective reporting is good for Google
a. Less resources spent to handle requests
b. Less issues to investigate
2. Effective reporting is good for you
a. Higher reliability
b. Faster reports generation
c. Save memory / CPU / bandwidth

6. Why this talk?
1. Effective reporting is good for Google
a. Less resources spent to handle requests
b. Less issues to investigate
2. Effective reporting is good for you
a. Higher reliability
b. Faster reports generation
c. Save memory / CPU / bandwidth
3. Effective reporting is good for other users
a. Saving resources means faster reporting for everyone

7. 1. Limit data volume

8. Limit data volume
1. Make sure dates range is relevant
Real world example:
SELECT <list of 10 columns>
FROM CAMPAIGN_PERFORMANCE_REPORT
DURING 20140709, 20151006
called every day.

9. Limit data volume (cont.)
2. Request only necessary columns
Real world example:
SELECT Impressions, ConversionRateManyPerClickSignificance, Period, UrlCustomParameters, ViewThroughConversionsSignificance, AccountCurrencyCode, AccountTimeZoneId,
AdvertiserExperimentSegmentationBin, AverageCpm, AverageTimeOnSite, AveragePosition, AdvertisingChannelType, Amount, ActiveViewImpressions, AdNetworkType1, AdNetworkType2,
AdvertisingChannelSubType, AverageCpc, AverageFrequency, AveragePageviews, AvgCostForOfflineInteraction, BiddingStrategyId, BiddingStrategyName, BiddingStrategyType, BidType,
BounceRate, BudgetId, CampaignId, CampaignName, CampaignStatus, ConversionRateManyPerClickSignificance, Period, UrlCustomParameters, ViewThroughConversionsSignificance,
AccountCurrencyCode, AccountDescriptiveName, AccountTimeZoneId, ActiveViewCpm, ActiveViewImpressions, AdNetworkType2, AdNetworkType1, AdvertisingChannelSubType,
AdvertiserExperimentSegmentationBin, AdvertisingChannelType, Amount, AverageCpc, AverageCpm, AverageFrequency, AveragePageviews, AveragePosition, AverageTimeOnSite,
AvgCostForOfflineInteraction, BiddingStrategyId, BiddingStrategyName, BiddingStrategyType, BidType, BounceRate, BudgetId, CampaignId, CampaignName, CampaignStatus,
ClickAssistedConversions, ClickAssistedConversionsOverLastClickConversions, ClickAssistedConversionValue, ClickConversionRate, ClickConversionRateSignificance, Clicks,
ClickSignificance, ClickType, ContentBudgetLostImpressionShare, ContentImpressionShare, ContentRankLostImpressionShare, ConversionCategoryName,
ConversionManyPerClickSignificance, ConversionRateManyPerClick, ConversionsManyPerClick, ConversionTrackerId, ConversionTypeName, ConversionValue, ConvertedClicks,
ConvertedClicksSignificance, Cost, CostPerConversionManyPerClick, CostPerConversionManyPerClickSignificance, CostPerConvertedClick, CostPerConvertedClickSignificance,
CostPerEstimatedTotalConversion, CostSignificance, CpcSignificance, CpmSignificance, Ctr, CtrSignificance, Date, DayOfWeek, CustomerDescriptiveName, Device,
EstimatedCrossDeviceConversions, EstimatedTotalConversionRate, EstimatedTotalConversions, EnhancedCpcEnabled, EstimatedTotalConversionValue,
EstimatedTotalConversionValuePerClick, EstimatedTotalConversionValuePerCost, ExternalCustomerId, HourOfDay, ImpressionAssistedConversions,
ImpressionAssistedConversionsOverLastClickConversions, ImpressionAssistedConversionValue, ImpressionReach, Impressions, ImpressionSignificance, InvalidClickRate, InvalidClicks,
IsBudgetExplicitlyShared, LabelIds, Labels, Month, MonthOfYear, NumOfflineImpressions, NumOfflineInteractions, OfflineInteractionCost, OfflineInteractionRate, PercentNewVisitors,
PrimaryCompanyName, PositionSignificance, Quarter, RelativeCtr, SearchBudgetLostImpressionShare, SearchExactMatchImpressionShare, SearchImpressionShare,
SearchRankLostImpressionShare, ServingStatus, Slot, TotalBudget, TotalCost, TrackingUrlTemplate, ValuePerConversionManyPerClick, ValuePerConvertedClick,
ValuePerEstimatedTotalConversion, ViewThroughConversions, Week, Year FROM CAMPAIGN_PERFORMANCE_REPORT during 20150901,20150930

10. Limit data volume (cont.)
2. Request only necessary columns
Real world example:
SELECT Impressions, ConversionRateManyPerClickSignificance, Period, UrlCustomParameters, ViewThroughConversionsSignificance, AccountCurrencyCode, AccountTimeZoneId,
AdvertiserExperimentSegmentationBin, AverageCpm, AverageTimeOnSite, AveragePosition, AdvertisingChannelType, Amount, ActiveViewImpressions, AdNetworkType1, AdNetworkType2,
AdvertisingChannelSubType, AverageCpc, AverageFrequency, AveragePageviews, AvgCostForOfflineInteraction, BiddingStrategyId, BiddingStrategyName, BiddingStrategyType, BidType,
BounceRate, BudgetId, CampaignId, CampaignName, CampaignStatus, ConversionRateManyPerClickSignificance, Period, UrlCustomParameters, ViewThroughConversionsSignificance,
AccountCurrencyCode, AccountDescriptiveName, AccountTimeZoneId, ActiveViewCpm, ActiveViewImpressions, AdNetworkType2, AdNetworkType1, AdvertisingChannelSubType,
AdvertiserExperimentSegmentationBin, AdvertisingChannelType, Amount, AverageCpc, AverageCpm, AverageFrequency, AveragePageviews, AveragePosition, AverageTimeOnSite,
AvgCostForOfflineInteraction, BiddingStrategyId, BiddingStrategyName, BiddingStrategyType, BidType, BounceRate, BudgetId, CampaignId, CampaignName, CampaignStatus,
ClickAssistedConversions, ClickAssistedConversionsOverLastClickConversions, ClickAssistedConversionValue, ClickConversionRate, ClickConversionRateSignificance, Clicks,
ClickSignificance, ClickType, ContentBudgetLostImpressionShare, ContentImpressionShare, ContentRankLostImpressionShare, ConversionCategoryName,
ConversionManyPerClickSignificance, ConversionRateManyPerClick, ConversionsManyPerClick, ConversionTrackerId, ConversionTypeName, ConversionValue, ConvertedClicks,
ConvertedClicksSignificance, Cost, CostPerConversionManyPerClick, CostPerConversionManyPerClickSignificance, CostPerConvertedClick, CostPerConvertedClickSignificance,
CostPerEstimatedTotalConversion, CostSignificance, CpcSignificance, CpmSignificance, Ctr, CtrSignificance, Date, DayOfWeek, CustomerDescriptiveName, Device,
EstimatedCrossDeviceConversions, EstimatedTotalConversionRate, EstimatedTotalConversions, EnhancedCpcEnabled, EstimatedTotalConversionValue,
EstimatedTotalConversionValuePerClick, EstimatedTotalConversionValuePerCost, ExternalCustomerId, HourOfDay, ImpressionAssistedConversions,
ImpressionAssistedConversionsOverLastClickConversions, ImpressionAssistedConversionValue, ImpressionReach, Impressions, ImpressionSignificance, InvalidClickRate, InvalidClicks,
IsBudgetExplicitlyShared, LabelIds, Labels, Month, MonthOfYear, NumOfflineImpressions, NumOfflineInteractions, OfflineInteractionCost, OfflineInteractionRate, PercentNewVisitors,
PrimaryCompanyName, PositionSignificance, Quarter, RelativeCtr, SearchBudgetLostImpressionShare, SearchExactMatchImpressionShare, SearchImpressionShare,
SearchRankLostImpressionShare, ServingStatus, Slot, TotalBudget, TotalCost, TrackingUrlTemplate, ValuePerConversionManyPerClick, ValuePerConvertedClick,
ValuePerEstimatedTotalConversion, ViewThroughConversions, Week, Year FROM CAMPAIGN_PERFORMANCE_REPORT during 20150901,20150930
AccountCurrencyCode
AccountTimeZoneId
AccountDescriptiveName

11. Limit data volume (cont.)
2. Request only necessary columns
Real world example:
SELECT Impressions, ConversionRateManyPerClickSignificance, Period, UrlCustomParameters, ViewThroughConversionsSignificance, AccountCurrencyCode, AccountTimeZoneId,
AdvertiserExperimentSegmentationBin, AverageCpm, AverageTimeOnSite, AveragePosition, AdvertisingChannelType, Amount, ActiveViewImpressions, AdNetworkType1, AdNetworkType2,
AdvertisingChannelSubType, AverageCpc, AverageFrequency, AveragePageviews, AvgCostForOfflineInteraction, BiddingStrategyId, BiddingStrategyName, BiddingStrategyType, BidType,
BounceRate, BudgetId, CampaignId, CampaignName, CampaignStatus, ConversionRateManyPerClickSignificance, Period, UrlCustomParameters, ViewThroughConversionsSignificance,
AccountCurrencyCode, AccountDescriptiveName, AccountTimeZoneId, ActiveViewCpm, ActiveViewImpressions, AdNetworkType2, AdNetworkType1, AdvertisingChannelSubType,
AdvertiserExperimentSegmentationBin, AdvertisingChannelType, Amount, AverageCpc, AverageCpm, AverageFrequency, AveragePageviews, AveragePosition, AverageTimeOnSite,
AvgCostForOfflineInteraction, BiddingStrategyId, BiddingStrategyName, BiddingStrategyType, BidType, BounceRate, BudgetId, CampaignId, CampaignName, CampaignStatus,
ClickAssistedConversions, ClickAssistedConversionsOverLastClickConversions, ClickAssistedConversionValue, ClickConversionRate, ClickConversionRateSignificance, Clicks,
ClickSignificance, ClickType, ContentBudgetLostImpressionShare, ContentImpressionShare, ContentRankLostImpressionShare, ConversionCategoryName,
ConversionManyPerClickSignificance, ConversionRateManyPerClick, ConversionsManyPerClick, ConversionTrackerId, ConversionTypeName, ConversionValue, ConvertedClicks,
ConvertedClicksSignificance, Cost, CostPerConversionManyPerClick, CostPerConversionManyPerClickSignificance, CostPerConvertedClick, CostPerConvertedClickSignificance,
CostPerEstimatedTotalConversion, CostSignificance, CpcSignificance, CpmSignificance, Ctr, CtrSignificance, Date, DayOfWeek, CustomerDescriptiveName, Device,
EstimatedCrossDeviceConversions, EstimatedTotalConversionRate, EstimatedTotalConversions, EnhancedCpcEnabled, EstimatedTotalConversionValue,
EstimatedTotalConversionValuePerClick, EstimatedTotalConversionValuePerCost, ExternalCustomerId, HourOfDay, ImpressionAssistedConversions,
ImpressionAssistedConversionsOverLastClickConversions, ImpressionAssistedConversionValue, ImpressionReach, Impressions, ImpressionSignificance, InvalidClickRate, InvalidClicks,
IsBudgetExplicitlyShared, LabelIds, Labels, Month, MonthOfYear, NumOfflineImpressions, NumOfflineInteractions, OfflineInteractionCost, OfflineInteractionRate, PercentNewVisitors,
PrimaryCompanyName, PositionSignificance, Quarter, RelativeCtr, SearchBudgetLostImpressionShare, SearchExactMatchImpressionShare, SearchImpressionShare,
SearchRankLostImpressionShare, ServingStatus, Slot, TotalBudget, TotalCost, TrackingUrlTemplate, ValuePerConversionManyPerClick, ValuePerConvertedClick,
ValuePerEstimatedTotalConversion, ViewThroughConversions, Week, Year FROM CAMPAIGN_PERFORMANCE_REPORT during 20150901,20150930
Labels and LabelIds
Month and MonthOfYear
AdNetworkType2, AdNetworkType1

12. Limit data volume (cont.)
3. Limit number of segmenting columns
Example 1: no segmenting variables (campaign performance report)
Campaign ID Impressions Clicks
262575047721 194201 1283

13. Limit data volume (cont.)
3. Limit number of segmenting columns
Example 2: 1 segmenting variable (campaign performance report)
Ad Network Campaign ID Impressions Clicks
Google search 262575047721 101113 737
Search partners 262575047721 93088 546
Display Network 262575047721 9288 311

14. Limit data volume (cont.)
3. Limit number of segmenting columns
Example 3: 2 segmenting variables (campaign performance report)
Date Ad Network Campaign ID Impressions Clicks
2015-09-01 Google search 262575047721 *** ***
2015-09-01 Search partners 262575047721 *** ***
2015-09-01 Display Network 262575047721 *** ***
2015-09-02 Search partners 262575047721 *** ***
2015-09-02 Google search 262575047721 *** ***
2015-09-02 Display Network 262575047721 *** ***
2015-09-03 Google search 262575047721 *** ***
…….. …….. …….. …….. ……..

15. Limit data volume (cont.)
3. Limit number of segmenting columns

16. 2. Know your data
freshness intervals

17. Data freshness
Click logged
Click
processed
Click stored
Click
happened
time

18. Data freshness (cont.)
time
clicks
3 am

19. Data freshness (cont.)
Click logged
Click
processed
Click stored
Click
happened
time
Invalid clicks
removal

20. Data freshness (cont.)
time
clicks
3 am t+3d-8d

21. Data freshness (cont.)
Takeaways:
● If you need billing data:
○ Wait at least 72 hours after the date to pull the report
● If you need live data:
○ Make sure you don’t treat numbers as final
● “Today’s data” is unpredictably delayed, polling is not useful
● No reason to re-fetch data again, if you already have final stats

22. Data freshness: links
The Source of Truth:
● https://support.google.com/adwords/answer/2544985
or
● Google for “adwords data freshness”
Metric/Report
When your data for Tuesday will be
ready if you’re in San Francisco
(PST).
When your data for Tuesday will
be ready if you’re in London
(GMT).
When your data for Tuesday will
be ready if you’re in Tokyo (JST).
Most metrics, including:
clicks, impressions, and
conversions
3 a.m. Wed. 3 a.m. Wed. 3 a.m. Wed.
Automatic placements,
geographic, and Search terms
6 a.m. Wed. 6 a.m. Wed. 6 a.m. Wed.
Auction insights 10 a.m. Wed. 8 p.m. Wed. 4 a.m. Thurs.
Impression share 1 p.m. Wed. 11 p.m. Wed. 7 a.m. Thurs.
Top movers 6 p.m. Wed. 2 a.m. Thurs. 10 a.m. Thurs.

23. 3. Utilize compression

24. Utilize compression
● Why use compression?
● Compression options available with AdWords API

25. ● Why use compression?
● Compression options available with AdWords API
Utilize compression (cont.)
Format Size % of XML size
XML 270’722 KB 100%
CSV 87’315 KB ~32%
GZIPPED_XML 22’059 KB ~8%
GZIPPED_CSV 18’648 KB ~7%

26. Utilize compression (cont.)
● Why use compression?
● Compression options available with AdWords API
a. Transparent compression
b. Explicit compression

27. Utilize compression (cont.)
● Why use compression?
● Compression options available with AdWords API
Format % of all (*) reporting requests
XML 8.48%
CSV 18.06%
GZIPPED_XML 5.92%
GZIPPED_CSV 65.16%
TSV 2.33%
CSVFOREXCEL 0.05%
(*) 90 days of requests explored

28. ● Why use compression?
● Compression options available with AdWords API
Utilize compression (cont.)
Transparent compression % of all (*) reporting requests
Enabled 68.2%
Disabled 31.8%
(*) 90 days of requests explored

29. Utilize compression (cont.)
● Explicit compression
○ Set format to GZIPPED_*
● Transparent compression
○ Client libraries
■ See library-specific configuration value
○ Custom libraries
■ Set HTTP headers:
User-agent Your Application name (gzip)
Accept-Encoding: gzip

30. 4. Employ multithreading

31. Employ multithreading
● Request multiple reports in parallel
○ Start around 10 threads and monitor for errors
● Split very large reports into multiple smaller ones
○ Yearly report to 12 monthly reports
● Avoid parallel report requests against a single CID
○ Access different sets of data concurrently instead
● Max threads number varies and depends on many factors
○ If concurrency is too high, rate exceeded error will be returned by the API

32. Employ multithreading (cont.)
● Make sure if one report failed, only this report is retried
● Share reporting errors data between threads
● Use dynamic threads count

33. 5. Fix error handling

34. Error handling -- stats
(*) 90 days of requests explored
Error class Error examples % of all (*) requests
with errors

35. Error handling -- stats
(*) 90 days of requests explored
Error class Error examples % of all (*) requests
with errors
Authentication issue USER_PERMISSION_DENIED
OAUTH_TOKEN_INVALID
OAUTH_TOKEN_REVOKED

36. Error handling -- stats
(*) 90 days of requests explored
Error class Error examples % of all (*) requests
with errors
Authentication issue USER_PERMISSION_DENIED
OAUTH_TOKEN_INVALID
OAUTH_TOKEN_REVOKED
User input issue INVALID_PREDICATE_FIELD_NAME
INVALID_FIELD_NAME_FOR_REPORT
NOT_ADS_USER

37. Error handling -- stats
(*) 90 days of requests explored
Error class Error examples % of all (*) requests
with errors
Authentication issue USER_PERMISSION_DENIED
OAUTH_TOKEN_INVALID
OAUTH_TOKEN_REVOKED
User input issue INVALID_PREDICATE_FIELD_NAME
INVALID_FIELD_NAME_FOR_REPORT
NOT_ADS_USER
Google error InternalApiError.*

38. Error handling -- stats
(*) 90 days of requests explored
Error class Error examples % of all (*) requests
with errors
Authentication issue USER_PERMISSION_DENIED
OAUTH_TOKEN_INVALID
OAUTH_TOKEN_REVOKED
75.9687%
User input issue INVALID_PREDICATE_FIELD_NAME
INVALID_FIELD_NAME_FOR_REPORT
NOT_ADS_USER
Google error InternalApiError.*

39. Error handling -- stats
(*) 90 days of requests explored
Error class Error examples % of all (*) requests
with errors
Authentication issue USER_PERMISSION_DENIED
OAUTH_TOKEN_INVALID
OAUTH_TOKEN_REVOKED
75.9687%
User input issue INVALID_PREDICATE_FIELD_NAME
INVALID_FIELD_NAME_FOR_REPORT
NOT_ADS_USER
24.0311%
Google error InternalApiError.*

40. Error handling -- stats
(*) 90 days of requests explored
Error class Error examples % of all (*) requests
with errors
Authentication issue USER_PERMISSION_DENIED
OAUTH_TOKEN_INVALID
OAUTH_TOKEN_REVOKED
75.9687%
User input issue INVALID_PREDICATE_FIELD_NAME
INVALID_FIELD_NAME_FOR_REPORT
NOT_ADS_USER
24.0311%
Google error InternalApiError.* 0.0002%

41. Error handling (cont.)
Takeaways:
● Make sure to collect error messages
● Classify errors for retriable and non-retriable
● Review your authentication tracking

42. Error handling (cont.)
Takeaways:
● Make sure to collect error messages
● Classify errors for retriable and non-retriable
● Review your authentication tracking
Common errors described in docs:
https://developers.google.com/adwords/api/docs/common-errors

43. 6. Time your requests

44. Time your requests
● Server-side resources are limited and shared
● Rate Exceeded error is returned when capacity is
exhausted
● Maximal rate depends on concurrent users, which
is mostly time of the day

45. Time your requests (cont.)

46. Time your requests (cont.)

47. Time your requests (cont.)

48. Time your requests (cont.)

49. Time your requests (cont.)
Takeaways:
● Running a job at a “good time” can save you ~5%
of execution time
● Best time to run reports is over the weekend
● Worst time to run is on Wed, 5th of the month at 10am
○ San Francisco time zone
● Google has great infrastructure to handle request
spikes

50. 7. Use library features

51. Use library features
1. Reporting utilities

52. Use library features
1. Reporting utilities
Response type Report size Use case
String Small Whole report loaded into memory
File Large For large reports, can be passed to another process
Stream Any Download in chunks, online processing

53. Use library features
1. Reporting utilities
2. Compression
Response type Report size Use case
String Small Whole report loaded into memory
File Large For large reports, can be passed to another process
Stream Any Download in chunks, online processing

54. Use library features
1. Reporting utilities
2. Compression
3. Reporting examples
Response type Report size Use case
String Small Whole report loaded into memory
File Large For large reports, can be passed to another process
Stream Any Download in chunks, online processing

55. 8. Store data locally

56. Store data locally
● Identify the scope of your data
● Retrieve when the data is ready
● Implement data retrieval efficiently
● Save data locally

57. Store locally (cont.)
● AwReporting
○ Open source tool to download and store
AdWords reporting data
○ https://github.com/googleads/aw-reporting

58. Store locally (cont.)
● Anything you find you need often and have to re-request, eg.:
○ Label IDs and names
○ Geo location IDs
○ Constant criteria types (mobile operator, vertical, platform IDs, ...)
○ User list IDs and names
○ Shared set IDs
○ Account settings (currency, timezone, …)

59. Bonus:
Authorization for reporting

60. Authorization for reporting
Jobs manager
Reporting job
Configuration file
read
refresh token
OAuth server
Reporting jobReporting jobReporting job
request
access token
API server
API request
report data

61. Authorization for reporting
Jobs manager
Reporting job
Configuration file
read
refresh token
OAuth server
Reporting jobReporting jobReporting job
request
access token
API server
API request
report data

62. Authorization for reporting (cont.)
Reporting job starts
Read configuration
Access token
expired?
Make reporting request
Refresh access token
Update configuration
Yes
No

63. Resources
● Common errors
○ https://developers.google.com/adwords/api/docs/common-errors
● Reporting reference
○ https://developers.google.com/adwords/api/docs/appendix/reports
● Client libraries
○ https://developers.google.com/adwords/api/docs/clientlibraries
● AwReporting
○ https://github.com/googleads/aw-reporting