Sometimes your fastest queries can cause the most problems. I will take you beyond the slow query optimization and instead zero in on the performance impacts surrounding the quantity of your datastore hits. Using real world examples dealing with datastores such as Elasticsearch, MySQL, and Redis, I will demonstrate how many fast queries can wreak just as much havoc as a few big slow ones. With each example I will make use of the simple tools available in Ruby to decrease and eliminate the need for these fast and seemingly innocuous datastore hits.

1. Cache is King
Get the Most Bang for Your Buck From Ruby

2. Site Reliability Engineer

4. Adding Indexes
Using SELECT statements
Batch Processing

5. Elasticsearch::Transport::Errors::GatewayTimeout 504
{
"statusCode": 200,
"took": "100ms"
}

6. Resque

8. Demo Time!

9. Quantity of Datastore Hits

11. The average company has...
60 thousand
assets
24 million
vulnerabilities?

12. MySQL
Elasticsearch
Cluster

13. Serialization

14. MySQL
Elasticsearch
Cluster
ActiveModelSerializers

15. module Beehive
module Serializers
class Vulnerability < ActiveModel::Serializer
attributes :id, :client_id, :created_at, :updated_at,
:priority, :details, :notes, :asset_id,
:solution_id, :owner_id, :ticket_id
end
end
end

16. 200 MILLION

17. 11 hours and counting...

20. (1.6ms)
(0.9ms)
(4.1ms)(5.2ms)
(5.2ms)
(1.3ms)
(3.1ms)
(2.9ms)
(2.2ms)
(4.9ms)
(6.0ms)
(0.3ms)
(1.6ms)
(0.9ms)
(2.2ms)
(3.0ms)
(2.1ms)
(1.3ms)
(2.1ms)
(8.1ms)
(1.4ms)

21. MySQL

22. Bulk Serialization

23. class BulkVulnerabilityCache
attr_accessor :vulnerabilities, :client, :vulnerability_ids
def initialize(vulns, client)
self.vulnerabilities = vulns
self.vulnerability_ids = vulns.map(&:id)
self.client = client
end
# MySQL Lookups
end

24. module Serializers
class Vulnerability
attr_accessor :vulnerability, :cache
def initialize(vuln, bulk_cache)
self.cache = bulk_cache
self.vulnerability = vuln
end
end
end
self.cache = bulk_cache

25. class Vulnerability
has_many :custom_fields
end

26. CustomField.where(:vulnerability_id => vuln.id)
cache.fetch('custom_fields', vuln.id)

27. The Result...
(pry)> vulns = Vulnerability.limit(300);
(pry)> Benchmark.realtime { vulns.each(&:serialize) }
=> 6.022452222998254
(pry)> Benchmark.realtime do
> BulkVulnerability.new(vulns, [], client).serialize
> end
=> 0.7267019419959979

28. Decrease in database hits
Individual Serialization:
Bulk Serialization:
2,100
7

29. 1k vulns 1k vulns 1k vulns
Vulnerability Batches

30. 1k vulns 1k vulns 1k vulns
Vulnerability Batches
7k 7

31. MySQL Queries
Bulk Serialization
Deployed

32. Bulk Serialization
Deployed
RDS CPU Utilization

33. Process
in Bulk

36. Elasticsearch
Cluster
+
Redis
MySQL
Vulnerabilities

37. Redis.get
Client 1
Index
Client 2
Index
Client 3 & 4
Index

38. indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

39. indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

40. (pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”)
DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234"
DEBUG -- : [Redis] call_time=1.07 ms
GET

43. client_indexes = Hash.new do |h, client_id|
h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”)
end

44. indexing_hashes = vuln_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{client_id}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end
client_indexes[hash[:client_id]],

45. 1 + 1 + 1
Client 1 Client 2 Client 3
1k 1k 1k

46. 1000x

47. 65% job
speed up

48. Local Cache

49. Redis

50. Process
in Bulk
Hash
Cache

51. Sharded Databases
CLIENT 1
CLIENT 2
CLIENT 3

52. Asset.with_shard(client_id).find(1)

53. {
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}
Sharding Configuration

55. Sharding Configuration Size
20 bytes 1kb 13kb

56. 285 Workers

57. 7.8 MB/second

58. ActiveRecord::Base.connection

59. (pry)> ActiveRecord::Base.connection
=> #<Octopus::Proxy:0x000055b38c697d10
@proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8

60. module Octopus
class Proxy
attr_accessor :proxy_config
delegate :current_shard, :current_shard=,
:current_slave_group, :current_slave_group=,
:shard_names, :shards_for_group, :shards, :sharded,
:config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false
end
end

61. Know your gems

62. Process
in Bulk
Framework
Cache
Hash
Cache

63. Avoid making datastore hits you don’t
need

64. User.where(:id => user_ids).each do |user|
# Lots of user processing
end

65. FALSE

66. (pry)> User.where(:id => [])
User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0
=> []

68. return unless user_ids.any?
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

69. (pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
(pry)> Benchmark.realtime do
> 10_000.times do
> next unless ids.any?
> User.where(:id => [])
> end
> end
=> 0.0006368421018123627

70. (pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
“Ruby is slow”Hitting the database is slow!

71. User.where(:id => user_ids).each do |user|
# Lots of user processing
end

72. User.where(:id => user_ids).each do |user|
# Lots of user processing
end
users = User.where(:id => user_ids).active.short.single

73. .none

74. (pry)> User.where(:id => []).active.tall.single
User Load (0.7ms) SELECT `users`.* FROM `users` WHERE 1=0 AND
`users`.`active` = 1 AND `users`.`short` = 0 AND `users`.`single` = 1
=> []
(pry)> User.none.active.tall.single
=> []
.none in action...

77. Logging
pry(main)> Rails.logger.level = 0
$ redis-cli monitor > commands-redis-2018-10-01.txt
pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)

78. Preventing useless
datastore hits

80. Report
Elasticsearch
MySQL Redis

81. (pry)> Report.blank_reports.count
=> 10805
(pry)> Report.active.count
=> 25842
(pry)> Report.average_asset_count
=> 1657
Investigating Existing Reports

82. Report
Elasticsearch
MySQL Redis

84. 10+ hrs

85. 3 hrs

86. Process
in Bulk
Framework
Cache
Database
Guards
Hash
Cache

87. Resque Workers
Redis

88. 45 workers
45 workers
45 workers

89. 70 workers
70 workers
70 workers

93. 48 MB
16 MB

94. Redis Requests
70 workers
100k
200k

96. ?

97. Resque
Throttling

98. Redis Requests
100k
200k

99. Redis Network Traffic
48MB
16MB

101. Process
in Bulk
Framework
Cache
Database
Guards
Remove
Datastore
Hits
Hash
Cache

102. Every
datastore hit
COUNTS

104. Questions

105. Contact
https://www.linkedin.com/in/mollystruve/
https://github.com/mstruve
@molly_struve
molly.struve@gmail.com