Presented at Lucene/Solr Revolution 2016

1. OCTOBER 11-14, 2016 • BOSTON, MA

2. Why is my Solr slow?
Mike Drob
Software Engineer, Cloudera

3. Who Am I?
●
Apache HTrace (Incubating) Committer
●
Software Engineer @ Cloudera
●
Middle School LEGO Robotics Mentor

4. Outline
●
The need for tracing
●
HTrace introduction
●
Examples with Solr
●
Finding the slow server!

5. Scenario
●
User complains that queries are “slow”
●
Caches?
●
Or faceting?
●
Or sorting?
●
Or...?

6. The State of the Art
Run a query with debug=timing:
"timing":{
"time":178.0,
"prepare":{"time":7.0,
"query":{"time":4.0}}
"process":{"time":160.0,
"query":{"time":118.0},
"facet":{"time":33.0}}

7. What is Apache HTrace
●
Distributed Tracing Library
●
Primitive Type is the “Span”
●
Unique ID
●
Source + Description
●
Start/Stop Time
●
Additional Metadata
●
See Also: Dapper, Zipkin, Wingtips

8. Reference Architecture
Application
Trace
Gen
Application
Trace
Gen
Application
Trace
Gen
Application
Trace
Gen
Collector

9. Example Architecture
Solr
trace
Solr
trace
Solr
trace
Solr
trace
htraced

10. How Do I Trace?
●
Can Trace Any “Unit Of Work”
●
Method Calls
●
Threads
●
RPCs doQuery();
try (TraceScope scope = tracer.newScope(“work”)) {
doQuery();
}

11. What Does It Look Like?
● Trace of q=*:* and resulting sub-queries
● Data modeled on debug=timing

12. Aside: Benefits over debug=timing
●
Separate time-lines for each server
●
Ability to mark annotations
●
Data stored for later analysis
●
Trace non-query requests
●
Trace calls to other systems

13. Something More Complex
techproducts/select?q=*:*&facet.field=popularity

14. And Other Commands

15. And Other Commands

16. Configuration
●
New section in solr.xml
<trace>
<str name="span.receiver.classes">org.apache.htrace.impl.HTracedSpanReceiver</str>
<str name="htraced.receiver.address">trace-server:9075</str>
<str name="sampler.classes">org.apache.htrace.core.ProbabilitySampler</str>
</trace>
●
Sampler Options
●
Advanced: Force Trace
… traceid=<64-bit> …
●
Advanced: Buffer Options

17. Performance
●
Each trace add ~25ns of overhead
●
Search handler can generate ~20 traces
●
Traces are buffered in memory
●
May consider multiple recievers

18. Future Work
●
Add tracing to clients (SolrJ)
●
Leverage MDC logging
●
Async Requests

19. Demo

20. Demo (Backup)
"responseHeader":{
"zkConnected":true,
"status":0,
"QTime":5251,
"params":{
"q":"*:*",
"indent":"on",
"wt":"json",
"_":"1476334957442"}},

21. Demo (Backup)

22. Demo (Backup)

23. Demo (Backup)

24. Demo (Backup)

25. Thank you
mdrob@apache.org
@mikhaildrob