Sometimes your fastest queries can cause the most problems. I will take you beyond the slow query optimization and instead zero in on the performance impacts surrounding the quantity of your datastore hits. Using real world examples dealing with datastores such as Elasticsearch, MySQL, and Redis, I will demonstrate how many fast queries can wreak just as much havoc as a few big slow ones. With each example I will make use of the simple tools available in Ruby to decrease and eliminate the need for these fast and seemingly innocuous datastore hits.

1. Cache is King
Get the Most Bang for Your Buck From Ruby

2. Site Reliability Engineer

4. Adding Indexes
Using SELECT statements
Batch Processing

5. Elasticsearch::Transport::Errors::GatewayTimeout 504
{
"statusCode": 200,
"took": "100ms"
}

6. Resque

8. Demo Time!

9. Quantity of Datastore Hits

11. The average company has...
60 thousand
assets
24 million
vulnerabilities?

12. MySQL
Elasticsearch
Cluster

13. Serialization

14. MySQL
Elasticsearch
Cluster
ActiveModelSerializers

15. module Beehive
module Serializers
class Vulnerability < ActiveModel::Serializer
attributes :id, :client_id, :created_at, :updated_at,
:priority, :details, :notes, :asset_id,
:solution_id, :owner_id, :ticket_id
end
end
end

16. 200 MILLION

17. 11 hours and counting...

20. (1.6ms)
(0.9ms)
(4.1ms)(5.2ms)
(5.2ms)
(1.3ms)
(3.1ms)
(2.9ms)
(2.2ms)
(4.9ms)
(6.0ms)
(0.3ms)
(1.6ms)
(0.9ms)
(2.2ms)
(3.0ms)
(2.1ms)
(1.3ms)
(2.1ms)
(8.1ms)
(1.4ms)

21. MySQL

22. Bulk Serialization

23. class BulkVulnerabilityCache
attr_accessor :vulnerabilities, :client, :vulnerability_ids
def initialize(vulns, client)
self.vulnerabilities = vulns
self.vulnerability_ids = vulns.map(&:id)
self.client = client
end
# MySQL Lookups
end

24. module Serializers
class Vulnerability
attr_accessor :vulnerability, :cache
def initialize(vuln, bulk_cache)
self.cache = bulk_cache
self.vulnerability = vuln
end
end
end
self.cache = bulk_cache

25. class Vulnerability
has_many :custom_fields
end

26. CustomField.where(:vulnerability_id => vuln.id)
cache.fetch('custom_fields', vuln.id)

27. The Result...
(pry)> vulns = Vulnerability.limit(300);
(pry)> Benchmark.realtime { vulns.each(&:serialize) }
=> 6.022452222998254
(pry)> Benchmark.realtime do
> BulkVulnerability.new(vulns, [], client).serialize
> end
=> 0.7267019419959979

28. Decrease in database hits
Individual Serialization:
Bulk Serialization:
2,100
7

29. 1k vulns 1k vulns 1k vulns
Vulnerability Batches

30. 1k vulns 1k vulns 1k vulns
Vulnerability Batches
7k 7

31. MySQL Queries
Bulk Serialization
Deployed

32. Bulk Serialization
Deployed
RDS CPU Utilization

33. Process
in Bulk

36. Elasticsearch
Cluster
+
Redis
MySQL
Vulnerabilities

37. Redis
Client 1
Index
Client 2
Index
Client 3 & 4
Index

38. (pry)> Redis.get(“elasticsearch_index_#{client_id}”)
DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234"
DEBUG -- : [Redis] call_time=1.07 ms
GET

41. client_indexes = Hash.new do |h, client_id|
h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”)
end

42. indexing_hashes = vuln_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{client_id}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end
client_indexes[hash[:client_id]],

43. 1 + 1 + 2
Client 1 Client 2 Client 3 & 4
1k 1k 1k

44. 1000x

45. 65% job
speed up

46. Local Cache

47. Redis

48. Process
in Bulk
Hash
Cache

49. Sharded Databases
CLIENT 1
CLIENT 2
CLIENT 3

50. Asset.with_shard(client_id).find(1)

51. {
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}
Sharding Configuration

53. Sharding Configuration Size
20 bytes 1kb 13kb

54. 285 Workers

55. 7.8 MB/second

56. ActiveRecord::Base.connection

57. (pry)> ActiveRecord::Base.connection
=> #<Octopus::Proxy:0x000055b38c697d10
@proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8

58. module Octopus
class Proxy
attr_accessor :proxy_config
delegate :current_shard, :current_shard=,
:current_slave_group, :current_slave_group=,
:shard_names, :shards_for_group, :shards, :sharded,
:config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false
end
end

59. Know your gems

60. Process
in Bulk
Framework
Cache
Hash
Cache

61. Avoid making datastore hits you don’t
need

62. User.where(:id => user_ids).each do |user|
# Lots of user processing
end

63. FALSE

64. (pry)> User.where(:id => [])
User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0
=> []

66. return unless user_ids.any?
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

67. (pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
(pry)> Benchmark.realtime do
> 10_000.times do
> next unless ids.any?
> User.where(:id => [])
> end
> end
=> 0.0006368421018123627

68. (pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
“Ruby is slow”Hitting the database is slow!

69. User.where(:id => user_ids).each do |user|
# Lots of user processing
end

70. User.where(:id => user_ids).each do |user|
# Lots of user processing
end
users = User.where(:id => user_ids).active.short.single

71. .none

72. (pry)> User.where(:id => []).active.tall.single
User Load (0.7ms) SELECT `users`.* FROM `users` WHERE 1=0 AND
`users`.`active` = 1 AND `users`.`short` = 0 AND `users`.`single` = 1
=> []
(pry)> User.none.active.tall.single
=> []
.none in action...

75. Logging
pry(main)> Rails.logger.level = 0
$ redis-cli monitor > commands-redis-2018-10-01.txt
pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)

76. Preventing useless
datastore hits

78. Report
Elasticsearch
MySQL Redis

79. (pry)> Report.blank_reports.count
=> 10805
(pry)> Report.active.count
=> 25842
(pry)> Report.average_asset_count
=> 1657
Investigating Existing Reports

80. Report
Elasticsearch
MySQL Redis

82. 10+ hrs

83. 3 hrs

84. Process
in Bulk
Framework
Cache
Database
Guards
Hash
Cache

85. Resque Workers
Redis

86. 45 workers
45 workers
45 workers

87. 70 workers
70 workers
70 workers

91. 48 MB
16 MB

92. Redis Requests
70 workers
100k
200k

94. ?

95. Resque
Throttling

96. Redis Requests
100k
200k

97. Redis Network Traffic
48MB
16MB

99. Process
in Bulk
Framework
Cache
Database
Guards
Remove
Datastore
Hits
Hash
Cache

100. Every
datastore hit
COUNTS

102. Questions

103. Contact
https://www.linkedin.com/in/mollystruve/
https://github.com/mstruve
@molly_struve
molly.struve@gmail.com