Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
How 
to 
find 
and 
fix 
your 
Java 
APEX 
ADF 
OBIEE 
.NET 
SQL 
PL/SQL 
application 
performance 
problem 
Cary 
Millsap...
@CaryMillsap 
2020 
2015 
Method R Trace 
2010 
2005 
2000 
1995 
System Performance Group 
Oracle APS 
1990 
1985 
100 
4...
@CaryMillsap 
Q What 
is 
the 
most 
common 
Oracle 
performance 
problem 
you 
see?” 
3 
“
@CaryMillsap 
What 
is 
the 
most 
common 
Oracle 
performance 
problem 
you 
see?” 
4 
“ 
Assuming 
that 
other 
people’s...
Java 
APEX 
ADF 
OBIEE 
.NET 
SQL 
PL/SQL 
@CaryMillsap 5
@CaryMillsap 
What 
is 
a 
performance problem? 
6
@@CCaarryyMMiillllssaapp 7
@CaryMillsap 
Performance is not 
an attribute of a system. 
8
#define FASTid (Rid ≤ SLRid) 
ID USERNAME OPERATION R SLR 
-- -------- --------- ----- --- 
1 FCHANG OE BOOK 2.019 2.0 
2 ...
#define FASTid (Rid ≤ SLRid) 
ID USERNAME OPERATION R SLR FAST? 
-- -------- --------- ----- --- ----- 
1 FCHANG OE BOOK 2...
@CaryMillsap 
Performance is an attribute of 
each individual experience 
with a system. 
11
EXPERIENCE 
• id 
• task-id 
• user-id 
• ip-address 
• start-time 
• end-time 
• ERROR-code 
• WORK-done 
TASK 
• id 
• n...
<experience 
id 
= 
"b3196c98-­‐906d-­‐4394-­‐bc55-­‐0339518a63b2" 
task-­‐id 
= 
"7" 
uid 
= 
"238" 
ip 
= 
"142.128.130....
@CaryMillsap 
click 
button 
link 
row 
query 
report 
{job} “My 
has to finish quickly.” 
14 
This 
is 
what 
performance...
@CaryMillsap 
click 
button 
link 
row 
query 
report 
{job} “My 
has to finish quickly.” 
15 
A 
performance 
problem 
is...
“How long does it take?” 
Response 
time 
(R) 
Duration 
from 
service 
request 
to 
service 
fulfillment. 
Sanjay Nancy K...
Two 
big 
questions... 
1. How 
long 
did 
it 
take? 
2. Why? 
“How long does it take?” 
Response 
time 
(R) 
Duration 
fr...
@CaryMillsap 
Method 
R 
18
@CaryMillsap 
1. Select 
the 
experience 
you 
need 
to 
improve. 
2. Measure 
its 
response 
time 
(R) 
in 
detail. 
3. E...
@CaryMillsap 
1. Select 
the 
experience 
you 
need 
to 
improve. 
2. Measure 
its 
response 
time 
(R) 
in 
detail. 
3. E...
Method 
R 
 (QEWUQPVJGTGCNCEVWCNRTQDNGO 
 %CVEJKVKPVJGCEV 
 QVJGUOCTVVJKPI 
 3WKVYJGPňJGNRKPIʼnUVQRUJGNRKPI 
@CaryMillsap 2...
MeTHOD R 
OPTIMIZE ANYTHING 
@CaryMillsap 22
@CaryMillsap 
1. Select 
the 
experience 
you 
need 
to 
improve. 
2. Measure 
its 
response 
time 
(R) 
in 
detail. 
3. E...
@CaryMillsap 
1. Select 
the 
experience 
you 
need 
to 
improve. 
2. Measure 
its 
response 
time 
(R) 
in 
detail. 
3. E...
@@CCaarryyMMiillllssaapp 25
@CaryMillsap 
Oracle 
extended 
SQL 
tracing 
D ATA B A S E EXADATA 
ENTERPRISE EDITION 
is 
a 
feature 
of 
D ATA B A S E...
@CaryMillsap 
Measuring 
Oracle 
response 
times 
27
❶ 
Activate 
tracing 
❷ 
Get 
the 
trace 
file 
❸ 
Understand 
its 
story 
@CaryMillsap 28
@CaryMillsap 
❶ 
Activate 
tracing 
❷ 
Get 
the 
trace 
file 
❸ 
Understand 
its 
story 
29
This 
is 
the 
hardest 
part. 
...But 
only 
the 
first 
time. 
After 
that, 
you 
just 
lather, 
rinse, 
repeat. 
@CaryMi...
https://app.com/apex/f?p=150:1:5547991082303::NO:::P_TRACE=YES 
@CaryMillsap 
31 
Well, 
it’s 
easy 
in 
Oracle 
APEX. 
To...
@CaryMillsap 
Other 
technologies 
require 
a 
little 
more 
work. 
First, 
the 
basics. 
32
dbms_monitor.session_trace_enable( 
session_id 
= 
null, 
serial_num 
= 
null, 
waits 
= 
true, 
binds 
= 
true, 
plan_sta...
if 
(should_trace('OE 
BOOK', 
dbms_random.value(0,1)) 
{ 
dbms_monitor.session_trace_enable( 
@CaryMillsap 
session_id 
=...
sub 
should_trace(task_name, 
r) 
{ 
select 
trace_proportion 
from 
trace_control 
where 
task_name 
= 
:t; 
return 
(r 
...
@CaryMillsap 
Oracle 
Database 
helps 
you 
implement 
run 
time 
tracing 
decisions... 
...without 
having 
to 
make 
you...
dbms_monitor.serv_mod_act_trace_enable( 
service_name 
= 
'SYS$USERS', 
module_name 
= 
'OE 
BOOK', 
action_name 
= 
dbms_...
How 
you 
set 
your 
module 
name 
varies 
by 
technology. 
@CaryMillsap 
SQL 
PL/SQL 
Java 
ADF 
.NET 
OBIEE 
38
dbms_application_info.set_module( 
module_name 
= 
'OE 
BOOK', 
action_name 
= 
sys_guid() 
); 
-­‐-­‐ 
Your 
‘book 
order...
String 
metrics[] 
= 
new 
String[OraCxn.END_TO_END_STATE_INDEX_MAX]; 
metrics[END_TO_END_MODULE_INDEX] 
= 
OE 
BOOK; 
met...
conn.ModuleName 
= 
OE 
BOOK; 
conn.ActionName 
= 
Guid.NewGuid().toString(); 
// 
Your 
‘book 
order’ 
code 
conn.ModuleN...
OBIEE 
To 
set 
your 
code’s 
module 
and 
action 
names... 
@CaryMillsap 42
@CaryMillsap 
Here’s 
the 
goal. 
43
User’s 
@CaryMillsap 
R 
experience 
Oracle 
trace 
file 
44 
User App Oracle DB 
time 
You 
want 
this 
to 
be 
small 
Yo...
Another 
experience 
@CaryMillsap 
An 
experience 
Not 
the 
trace 
file 
you 
want 
45 
User App Oracle DB 
time
Another 
experience 
@CaryMillsap 
An 
experience 
You 
want 
one 
trace 
file 
per 
experience 
46 
User App Oracle DB 
t...
@CaryMillsap 
The 
goal: 
Trace 
exactly 
each 
user 
experience 
you 
care 
about. 
...So 
that 
you 
can 
see 
how 
your...
@@CCaarryyMMiillllssaapp 48
@CaryMillsap 
This 
is 
what 
you’re 
looking 
at 
when 
you 
use 
systemwide 
aggregations. 
49 
User App Oracle DB 
time
❶ 
Activate 
tracing 
❷ 
Get 
the 
trace 
file 
❸ 
Understand 
its 
story 
@CaryMillsap 50
This 
is 
the 
boring 
part. 
...But 
it’s 
an 
inexpensive 
problem 
to 
solve. 
@CaryMillsap 51
@CaryMillsap 
Some 
things 
to 
know... 
Your 
trace 
file 
is 
on 
the 
Oracle 
Database 
server, 
in 
the 
diagnostic_de...
Please, will you help me find my trace file? 
@CaryMillsap 53
@CaryMillsap 
There 
are 
lots 
of 
ways 
to 
fetch 
the 
trace 
data. 
FTP 
Samba 
NFS 
mount 
portable 
disk 
USB 
thumb...
Fn’m [ mifp_^ jlif_g. 
@CaryMillsap 
Fetching 
trace 
files 
can 
be 
easy. 
You 
can 
build 
tools, 
or 
you 
can 
buy 
t...
❶ 
Activate 
tracing 
❷ 
Get 
the 
trace 
file 
❸ 
Understand 
its 
story 
@CaryMillsap 56
This 
is 
the 
FUN 
part. 
@CaryMillsap 57
What’s 
in 
there?! 
@CaryMillsap 58
An 
Oracle 
trace 
file 
is 
a 
log 
that 
shows 
what 
your 
code 
did 
inside 
the 
Oracle 
Database. 
@CaryMillsap 59
@CaryMillsap 
Some 
things 
to 
know... 
Oracle 
writes 
a 
trace 
line 
when 
a 
call 
(db|os) 
finishes. 
There 
are 
tw...
For 
more 
details... 
method-­‐r.com/papers 
1. 
Mastering 
Performance 
with 
Extended 
SQL 
Trace 
2. 
For 
Developers:...
@CaryMillsap 
Let’s 
look 
at 
some 
trace 
lines... 
62
Oracle 
kernel 
code 
path 
begin prepare 
CPU 
latch-related syscall 
CPU 
end prepare 
begin exec 
CPU 
write(SQLNET_OUT...
Oracle 
extended 
SQL 
trace 
data 
WAIT #42: nam='latch: library cache'… 
PARSE #42:c=10000,… 
WAIT #42: nam='SQL*Net mes...
Oracle 
extended 
SQL 
trace 
data 
WAIT #42: nam='latch: library cache'… 
PARSE #42:c=10000,… 
WAIT #42: nam='SQL*Net mes...
Oracle 
extended 
SQL 
trace 
data 
WAIT #42: nam='latch: library cache'… 
PARSE #42:c=10000,… 
WAIT #42: nam='SQL*Net mes...
Oracle 
extended 
SQL 
trace 
data 
WAIT #42: nam='latch: library cache'… 
PARSE #42:c=10000,… 
WAIT #42: nam='SQL*Net mes...
@CaryMillsap 
WAIT #42: nam='latch: library cache'… 
PARSE #42:c=10000,… 
WAIT #42: nam='SQL*Net message to client'… 
EXEC...
@CaryMillsap 
There 
are 
lots 
of 
ways 
to 
summarize 
a 
trace 
file. 
tkprof 
SQL 
Developer 
[Trace] 
Viewer 
Trace 
...
Fn’m [ mifp_^ jlif_g. 
@CaryMillsap 
Profiling 
trace 
files 
can 
be 
easy. 
You 
can 
build 
tools, 
or 
you 
can 
buy 
...
@CaryMillsap 
What 
you 
can 
do 
with 
trace 
files 
71
@CaryMillsap 
Example 
1 
72
mrskew 
r1-­‐fixed.trc 
CALL-­‐NAME 
DURATION 
% 
CALLS 
MEAN 
MIN 
MAX 
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-...
mrskew 
-­‐-­‐group=($sqlid=~/^#/?:[.$sqlid.]) 
-­‐-­‐gl=SQLID 
-­‐-­‐name=message 
from 
client 
r1-­‐fixed.trc 
SQLID 
D...
mrskew 
-­‐-­‐rc=p10 
-­‐-­‐name=SQL*Net 
message 
from 
client 
r1-­‐fixed.trc 
RANGE 
{min 
≤ 
e 
 
max} 
DURATION 
% 
C...
@CaryMillsap 
App Oracle DB 
~.648 
s ~.650 
s 
time 
76 
~.001 
s 
~.001 
s 
Each 
SQL*Net 
message 
from 
client 
call 
...
mrskew 
-­‐-­‐name=dbcall 
-­‐-­‐select=$row 
-­‐-­‐slabel=ROWS 
-­‐-­‐precision=0 
r1-­‐fixed.trc 
CALL-­‐NAME 
ROWS 
% 
...
mrskew 
r1-­‐fixed.trc 
CALL-­‐NAME 
DURATION 
% 
CALLS 
MEAN 
MIN 
MAX 
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-...
@CaryMillsap 
Example 
2 
79
mrskew 
-­‐-­‐top=10 
prd1_ora_9031.trc 
CALL-­‐NAME 
DURATION 
% 
CALLS 
MEAN 
MIN 
MAX 
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­...
mrskew 
-­‐-­‐rc=p10 
-­‐-­‐name=parse 
prd1_ora_9031.trc 
RANGE 
{min 
≤ 
e 
 
max} 
DURATION 
% 
CALLS 
MEAN 
MIN 
MAX 
...
mrskew 
-­‐-­‐name=parse 
-­‐-­‐group=$sqlid 
-­‐-­‐gl=SQLID 
-­‐-­‐top=10 
-­‐-­‐sort=4nd 
prd1_ora_9031.trc 
SQLID 
DURA...
mrskew 
-­‐-­‐rc=ssqlid 
prd1_ora_9031.trc 
SSQLID 
DISTINCT-­‐TEXTS 
% 
CALLS 
MEAN 
MIN 
MAX 
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­...
mrskew 
-­‐-­‐top=10 
prd1_ora_9031.trc 
CALL-­‐NAME 
DURATION 
% 
CALLS 
MEAN 
MIN 
MAX 
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­...
@CaryMillsap 
You 
might 
have 
known 
that 
you 
should 
“use 
bind 
variables,” 
but 
you 
couldn’t 
have 
quantified 
t...
BASELINE: 
for 
each 
invoice 
number 
{ 
cursor 
= 
parse(“select 
...where 
invoice_number 
= 
” 
. 
number); 
exec(curs...
BASELINE: 
@CaryMillsap 
BAD 
for 
each 
invoice 
number 
{ 
cursor 
= 
parse(“select 
...where 
invoice_number 
= 
(” 
. ...
FIX 
1 
“Hey, 
let’s 
use 
bind 
variables”: 
@CaryMillsap 
STILL 
BAD 
for 
each 
invoice 
number 
{ 
cursor 
= 
parse(“s...
FIX 
2: 
@CaryMillsap 
BETTER 
cursor 
= 
parse(“select 
...where 
invoice_number 
= 
:a1)”); 
for 
each 
invoice 
number ...
@CaryMillsap 
And 
so 
on... 
90
@CaryMillsap 
Bad 
SQL 
Bad 
PL/SQL 
Slow 
network 
Missing 
indexes 
Parsing 
in 
a 
loop 
Hot 
block 
problems 
Not 
eno...
@CaryMillsap 
There 
are 
only 
two 
possible 
root 
causes 
for 
any 
response 
time 
problem: 
❶ 
Call 
count 
is 
too 
...
mrskew 
-­‐-­‐top=10 
prd1_ora_9031.trc 
CALL-­‐NAME 
DURATION 
% 
CALLS 
MEAN 
MIN 
MAX 
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­...
@CaryMillsap 
With 
a 
good 
trace 
file, 
you 
can 
predict 
the 
response 
time 
impact 
of 
a 
proposed 
change.* 
*Thi...
It 
just 
takes 
practice. 
@CaryMillsap 95
@CaryMillsap 
Conclusion 
96
@CaryMillsap 
Your 
code 
does 
stuff. 
Including 
some 
stuff 
inside 
Oracle. 
The 
time 
this 
stuff 
takes 
is 
your 
...
@CaryMillsap 
References 
98
@CaryMillsap 
Robyn 
Sands, 
et 
al. 
2010. 
Expert 
Oracle 
Practices. 
Apress 
Detailed 
information 
about 
instrumenti...
QA 
method-­‐r.com 
www.enkitec.com 
method-­‐r.com/facebook 
facebook.com/enkitec 
@MethodR 
@Enkitec 
cary.millsap@metho...
Prochain SlideShare
Chargement dans…5
×

How to find and fix your Oracle application performance problem

3 777 vues

Publié le

How long does your code take to run? Is it changing? When it is slow, WHY is it slow? Is it your fault, or somebody else's? Can you prove it? How much faster could your code be? Do you know how to measure the performance of your code as user workloads and data volumes increase? These are fundamental questions about performance, but the vast majority of Oracle application developers can't answer them. The most popular performance tools available to them—and to the database administrators that run their code in production—are incapable of answering any of these questions. But the Oracle Database can give you exactly what you need to answer these questions and many more. You can know exactly where YOUR CODE is spending YOUR TIME. This session explains how.

Publié dans : Logiciels
  • People used to laugh at me behind my back before I was in shape or successful. Once I lost a lot of weight, I was so excited that I opened my own gym, and began helping others. I began to get quite a large following of students, and finally, I didn't catch someone laughing at me behind my back any longer. CLICK HERE NOW ★★★ https://tinyurl.com/1minweight4u
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

How to find and fix your Oracle application performance problem

  1. 1. How to find and fix your Java APEX ADF OBIEE .NET SQL PL/SQL application performance problem Cary Millsap Method R Corporation @CaryMillsap ·∙ cary.millsap@method-­‐r.com DOUG Tech Day ·∙ Richardson, Texas 12:00n–1:45p Saturday 18 October 2014 © 2006, 2014 Method R Corporation 1 TM MeTHOD R
  2. 2. @CaryMillsap 2020 2015 Method R Trace 2010 2005 2000 1995 System Performance Group Oracle APS 1990 1985 100 45 4 TM MeTHOD Rhotsos Optimal Flexible Architecture Method R Profiler Method R Tools 2 Cary Millsap
  3. 3. @CaryMillsap Q What is the most common Oracle performance problem you see?” 3 “
  4. 4. @CaryMillsap What is the most common Oracle performance problem you see?” 4 “ Assuming that other people’s common problems must be your problem. ... QA
  5. 5. Java APEX ADF OBIEE .NET SQL PL/SQL @CaryMillsap 5
  6. 6. @CaryMillsap What is a performance problem? 6
  7. 7. @@CCaarryyMMiillllssaapp 7
  8. 8. @CaryMillsap Performance is not an attribute of a system. 8
  9. 9. #define FASTid (Rid ≤ SLRid) ID USERNAME OPERATION R SLR -- -------- --------- ----- --- 1 FCHANG OE BOOK 2.019 2.0 2 RSMITH OE SHIP 3.528 5.0 3 DJOHNSON OE PICK 1.211 5.0 4 FFORBES OE BOOK 0.716 2.5 5 FCHANG OE BOOK 1.917 2.5 6 LBUMONT PA MTCH 1.305 2.0 @CaryMillsap 9
  10. 10. #define FASTid (Rid ≤ SLRid) ID USERNAME OPERATION R SLR FAST? -- -------- --------- ----- --- ----- 1 FCHANG OE BOOK 2.019 2.0 N 2 RSMITH OE SHIP 3.528 5.0 Y 3 DJOHNSON OE PICK 1.211 5.0 Y 4 FFORBES OE BOOK 0.716 2.5 Y 5 FCHANG OE BOOK 1.917 2.5 Y 6 LBUMONT PA MTCH 1.305 2.0 Y @CaryMillsap 10
  11. 11. @CaryMillsap Performance is an attribute of each individual experience with a system. 11
  12. 12. EXPERIENCE • id • task-id • user-id • ip-address • start-time • end-time • ERROR-code • WORK-done TASK • id • name • ... SQL • ID • Task-id • ... N 1 1 N @CaryMillsap 12
  13. 13. <experience id = "b3196c98-­‐906d-­‐4394-­‐bc55-­‐0339518a63b2" task-­‐id = "7" uid = "238" ip = "142.128.130.186" t0 = "2014-­‐04-­‐10T08:32:14.137886" t1 = "2014-­‐04-­‐10T08:32:17.891173" err = "" work = "3" /> @CaryMillsap 13
  14. 14. @CaryMillsap click button link row query report {job} “My has to finish quickly.” 14 This is what performance is.
  15. 15. @CaryMillsap click button link row query report {job} “My has to finish quickly.” 15 A performance problem is when it doesn’t.
  16. 16. “How long does it take?” Response time (R) Duration from service request to service fulfillment. Sanjay Nancy Ken Jorge R t0 t1 R = t1 – t0 Two big questions... 1. How long did it take? 2. Why? @CaryMillsap 16
  17. 17. Two big questions... 1. How long did it take? 2. Why? “How long does it take?” Response time (R) Duration from service request to service fulfillment. Sanjay Nancy Ken Jorge R t0 t1 R = t1 – t0 @CaryMillsap 17
  18. 18. @CaryMillsap Method R 18
  19. 19. @CaryMillsap 1. Select the experience you need to improve. 2. Measure its response time (R) in detail. 3. Execute the best net-­‐payoff remedy. 4. Repeat until economically optimal. 19 Method R
  20. 20. @CaryMillsap 1. Select the experience you need to improve. 2. Measure its response time (R) in detail. 3. Execute the best net-­‐payoff remedy. 4. Repeat until economically optimal. 20 Method R
  21. 21. Method R (QEWUQPVJGTGCNCEVWCNRTQDNGO %CVEJKVKPVJGCEV QVJGUOCTVVJKPI 3WKVYJGPňJGNRKPIʼnUVQRUJGNRKPI @CaryMillsap 21
  22. 22. MeTHOD R OPTIMIZE ANYTHING @CaryMillsap 22
  23. 23. @CaryMillsap 1. Select the experience you need to improve. 2. Measure its response time (R) in detail. 3. Execute the best net-­‐payoff remedy. 4. Repeat until economically optimal. 23 Method R
  24. 24. @CaryMillsap 1. Select the experience you need to improve. 2. Measure its response time (R) in detail. 3. Execute the best net-­‐payoff remedy. 4. Repeat until economically optimal. 24 Method R How do you do this, when the it is your code?
  25. 25. @@CCaarryyMMiillllssaapp 25
  26. 26. @CaryMillsap Oracle extended SQL tracing D ATA B A S E EXADATA ENTERPRISE EDITION is a feature of D ATA B A S E STANDARD EDITION D ATA B A S E EXPRESS EDITION every Oracle Database. 26 Oracle7 1992 Oracle8 1997 Oracle8i 2000 Oracle9i 2001 Oracle10g 2004 Oracle11g 2007 Oracle 12c 2013
  27. 27. @CaryMillsap Measuring Oracle response times 27
  28. 28. ❶ Activate tracing ❷ Get the trace file ❸ Understand its story @CaryMillsap 28
  29. 29. @CaryMillsap ❶ Activate tracing ❷ Get the trace file ❸ Understand its story 29
  30. 30. This is the hardest part. ...But only the first time. After that, you just lather, rinse, repeat. @CaryMillsap 30
  31. 31. https://app.com/apex/f?p=150:1:5547991082303::NO:::P_TRACE=YES @CaryMillsap 31 Well, it’s easy in Oracle APEX. To decide at run time whether to trace your code...
  32. 32. @CaryMillsap Other technologies require a little more work. First, the basics. 32
  33. 33. dbms_monitor.session_trace_enable( session_id = null, serial_num = null, waits = true, binds = true, plan_stat = 'ALL_EXECUTIONS' ); -­‐-­‐ Your ‘book order’ code dbms_monitor.session_trace_disable( session_id = null, serial_num = null ); @CaryMillsap 33 To decide at compile time to trace all your code...
  34. 34. if (should_trace('OE BOOK', dbms_random.value(0,1)) { dbms_monitor.session_trace_enable( @CaryMillsap session_id = null, serial_num = null, waits = true, binds = true, plan_stat = 'ALL_EXECUTIONS' ); } -­‐-­‐ Your ‘book order’ code dbms_monitor.session_trace_disable( session_id = null, serial_num = null ); 34 To decide at run time whether to trace your code...
  35. 35. sub should_trace(task_name, r) { select trace_proportion from trace_control where task_name = :t; return (r = trace_proportion); } @CaryMillsap 35 ...where should_trace looks like this. task_name trace_proportion OE BOOK 0.05 OE PICK 0.02 OE SHIP 1.00 OE INVOICE 0.01 should_trace(“OE BOOK”, 0.00) → true should_trace(“OE BOOK”, 0.01) → true should_trace(“OE BOOK”, 0.02) → true ... should_trace(“OE BOOK”, 0.05) → true should_trace(“OE BOOK”, 0.06) → false should_trace(“OE BOOK”, 0.07) → false should_trace(“OE BOOK”, 0.08) → false ... should_trace(“OE BOOK”, 1.00) → false 5% 95% trace_control
  36. 36. @CaryMillsap Oracle Database helps you implement run time tracing decisions... ...without having to make your developers do the if block stuff. 36
  37. 37. dbms_monitor.serv_mod_act_trace_enable( service_name = 'SYS$USERS', module_name = 'OE BOOK', action_name = dbms_monitor.all_actions, waits = true, binds = true, plan_stat = 'ALL_EXECUTIONS' ); @CaryMillsap 37 The DBA does this, at run time. But this works only if your code sets its module name to “OE BOOK”.
  38. 38. How you set your module name varies by technology. @CaryMillsap SQL PL/SQL Java ADF .NET OBIEE 38
  39. 39. dbms_application_info.set_module( module_name = 'OE BOOK', action_name = sys_guid() ); -­‐-­‐ Your ‘book order’ code dbms_application_info.set_module( module_name = null, action_name = null ); @CaryMillsap 39 SQL PL/SQL To set your code’s module and action names...
  40. 40. String metrics[] = new String[OraCxn.END_TO_END_STATE_INDEX_MAX]; metrics[END_TO_END_MODULE_INDEX] = OE BOOK; metrics[END_TO_END_ACTION_INDEX] = UUID.randomUUID().toString(); conn.setEndToEndMetrics(metrics, (short) 0); // Your ‘book order’ code metrics[END_TO_END_MODULE_INDEX] = ; metrics[END_TO_END_ACTION_INDEX] = ; conn.setEndToEndMetrics(metrics, (short) 0); @CaryMillsap 40 Java ADF To set your code’s module and action names...
  41. 41. conn.ModuleName = OE BOOK; conn.ActionName = Guid.NewGuid().toString(); // Your ‘book order’ code conn.ModuleName = ; conn.ActionName = ; @CaryMillsap 41 ODP.NET To set your code’s module and action names...
  42. 42. OBIEE To set your code’s module and action names... @CaryMillsap 42
  43. 43. @CaryMillsap Here’s the goal. 43
  44. 44. User’s @CaryMillsap R experience Oracle trace file 44 User App Oracle DB time You want this to be small You want this to be small
  45. 45. Another experience @CaryMillsap An experience Not the trace file you want 45 User App Oracle DB time
  46. 46. Another experience @CaryMillsap An experience You want one trace file per experience 46 User App Oracle DB time
  47. 47. @CaryMillsap The goal: Trace exactly each user experience you care about. ...So that you can see how your code consumes time when it behaves properly, and when it misbehaves. 47
  48. 48. @@CCaarryyMMiillllssaapp 48
  49. 49. @CaryMillsap This is what you’re looking at when you use systemwide aggregations. 49 User App Oracle DB time
  50. 50. ❶ Activate tracing ❷ Get the trace file ❸ Understand its story @CaryMillsap 50
  51. 51. This is the boring part. ...But it’s an inexpensive problem to solve. @CaryMillsap 51
  52. 52. @CaryMillsap Some things to know... Your trace file is on the Oracle Database server, in the diagnostic_dest directory. Your file is probably called dbname_ora_spid_id.trc, where dbname is your db_name parameter value, spid is your session’s v$process.spid value, and id is your session’s tracefile_identifier value. Sessions with DOP = k can create 2k + 1 trace files. 52
  53. 53. Please, will you help me find my trace file? @CaryMillsap 53
  54. 54. @CaryMillsap There are lots of ways to fetch the trace data. FTP Samba NFS mount portable disk USB thumb drive Oracle Database directory objects Method R Trace extension for Oracle SQL Developer 3 54
  55. 55. Fn’m [ mifp_^ jlif_g. @CaryMillsap Fetching trace files can be easy. You can build tools, or you can buy them. 55
  56. 56. ❶ Activate tracing ❷ Get the trace file ❸ Understand its story @CaryMillsap 56
  57. 57. This is the FUN part. @CaryMillsap 57
  58. 58. What’s in there?! @CaryMillsap 58
  59. 59. An Oracle trace file is a log that shows what your code did inside the Oracle Database. @CaryMillsap 59
  60. 60. @CaryMillsap Some things to know... Oracle writes a trace line when a call (db|os) finishes. There are two primary line formats: one for db calls, one for os calls. Each call is associated with a SQL or PL/SQL statement through a cursor id. Each line contains a time stamp (tim) and a duration (e|ela). R ≠ Σ(e|ela) because parent call durations include child call durations. 60
  61. 61. For more details... method-­‐r.com/papers 1. Mastering Performance with Extended SQL Trace 2. For Developers: Making Friends with the Oracle Database @CaryMillsap 61
  62. 62. @CaryMillsap Let’s look at some trace lines... 62
  63. 63. Oracle kernel code path begin prepare CPU latch-related syscall CPU end prepare begin exec CPU write(SQLNET_OUT, result_to_client); end exec read(SQLNET_IN, next_request_from_client); begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); This is the kind of stuff your code causes the Oracle kernel to do. @CaryMillsap 63
  64. 64. Oracle extended SQL trace data WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… Oracle kernel code path begin prepare CPU latch-related syscall CPU end prepare begin exec CPU write(SQLNET_OUT, result_to_client); end exec read(SQLNET_IN, next_request_from_client); begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client); This is the kind of trace data your code produces. end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); @CaryMillsap 64
  65. 65. Oracle extended SQL trace data WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… Of course, you don’t directly get to see the kernel code path. @CaryMillsap 65
  66. 66. Oracle extended SQL trace data WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… ...Or that helpful grid that I drew for you. @CaryMillsap 66
  67. 67. Oracle extended SQL trace data WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… All you get to see is this. @CaryMillsap 67
  68. 68. @CaryMillsap WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… 68 Oracle kernel code path Oracle extended SQL trace data begin prepare CPU latch-related syscall CPU end prepare begin exec CPU write(SQLNET_OUT, result_to_client); end exec read(SQLNET_IN, next_request_from_client); begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); You can learn to envision the kernel’s code path that motivated your trace file.
  69. 69. @CaryMillsap There are lots of ways to summarize a trace file. tkprof SQL Developer [Trace] Viewer Trace Analyzer tvdxstat xtrace OraSRP Method R Profiler 69
  70. 70. Fn’m [ mifp_^ jlif_g. @CaryMillsap Profiling trace files can be easy. You can build tools, or you can buy them. 70
  71. 71. @CaryMillsap What you can do with trace files 71
  72. 72. @CaryMillsap Example 1 72
  73. 73. mrskew r1-­‐fixed.trc CALL-­‐NAME DURATION % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ SQL*Net message from client 1,403.927942 99.7% 2,161 0.649666 0.000000 0.927028 FETCH 3.013549 0.2% 2,161 0.001395 0.000000 0.005000 direct path read temp 1.259022 0.1% 83 0.015169 0.003287 0.046968 SQL*Net more data to client 0.141213 0.0% 2,460 0.000057 0.000005 0.001269 SQL*Net message to client 0.007964 0.0% 2,161 0.000004 0.000001 0.000376 -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ TOTAL (5) 1,408.349690 100.0% 9,026 0.156033 0.000000 0.927028 99.7% of the time is 2,161 network round-­‐trips. What SQL statements cause the round-­‐trips? @CaryMillsap 73
  74. 74. mrskew -­‐-­‐group=($sqlid=~/^#/?:[.$sqlid.]) -­‐-­‐gl=SQLID -­‐-­‐name=message from client r1-­‐fixed.trc SQLID DURATION % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ [7d0bv6ds85q1f] 1,403.927942 100.0% 2,161 0.649666 0.000000 0.927028 -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ TOTAL (1) 1,403.927942 100.0% 2,161 0.649666 0.000000 0.927028 Just one. All 2,161 round-­‐trips are executed on behalf of just one SQL statement. @CaryMillsap 74
  75. 75. mrskew -­‐-­‐rc=p10 -­‐-­‐name=SQL*Net message from client r1-­‐fixed.trc RANGE {min ≤ e max} DURATION % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ 1. 0.000000 0.000001 0.000000 0.0% 1 0.000000 0.000000 0.000000 2. 0.000001 0.000010 3. 0.000010 0.000100 4. 0.000100 0.001000 5. 0.001000 0.010000 6. 0.010000 0.100000 7. 0.100000 1.000000 1,403.927942 100.0% 2,160 0.649967 0.547110 0.927028 8. 1.000000 10.000000 9. 10.000000 100.000000 10. 100.000000 1,000.000000 11. 1,000.000000 +∞ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ TOTAL (11) 1,403.927942 100.0% 2,161 0.649666 0.000000 0.927028 Each round-­‐trip consumes an average of .649967 ≈ .650 s. Why? @CaryMillsap 75
  76. 76. @CaryMillsap App Oracle DB ~.648 s ~.650 s time 76 ~.001 s ~.001 s Each SQL*Net message from client call (~.650 s) looks like this. If round-­‐trip network latency is ~.002 s, then this experience is spending ~.648 s in the Java code executed between database calls.
  77. 77. mrskew -­‐-­‐name=dbcall -­‐-­‐select=$row -­‐-­‐slabel=ROWS -­‐-­‐precision=0 r1-­‐fixed.trc CALL-­‐NAME ROWS % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐ -­‐-­‐-­‐ -­‐-­‐-­‐ FETCH 216,017 100.0% 2,161 100 17 100 -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐ -­‐-­‐-­‐ -­‐-­‐-­‐ TOTAL (1) 216,017 100.0% 2,161 100 17 100 One final check... The trace file shows that the application, at least, is fetching an average of 100 rows per fetch call (per round-­‐trip). This helps explain the Java-­‐side latency, but still, .648 s to process just 100 rows needs some explaining. @CaryMillsap 77
  78. 78. mrskew r1-­‐fixed.trc CALL-­‐NAME DURATION % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ SQL*Net message from client 1,403.927942 99.7% 2,161 0.649666 0.000000 0.927028 FETCH 3.013549 0.2% 2,161 0.001395 0.000000 0.005000 direct path read temp 1.259022 0.1% 83 0.015169 0.003287 0.046968 SQL*Net more data to client 0.141213 0.0% 2,460 0.000057 0.000005 0.001269 SQL*Net message to client 0.007964 0.0% 2,161 0.000004 0.000001 0.000376 -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ TOTAL (5) 1,408.349690 100.0% 9,026 0.156033 0.000000 0.927028 No matter how long you try to“fix the database” here, you’re going to see at most only a .3% difference in response time. The problem here is in the Java. @CaryMillsap 78
  79. 79. @CaryMillsap Example 2 79
  80. 80. mrskew -­‐-­‐top=10 prd1_ora_9031.trc CALL-­‐NAME DURATION % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ PARSE 735.426197 78.9% 698 1.053619 0.000000 4.498316 SQL*Net message from client 104.762229 11.2% 1,378 0.076025 0.000391 3.554818 FETCH 91.800028 9.8% 680 0.135000 0.000000 0.506923 db file sequential read 0.104670 0.0% 14 0.007476 0.001067 0.016408 EXEC 0.083988 0.0% 349 0.000241 0.000000 0.002000 gc cr block 2-­‐way 0.073233 0.0% 96 0.000763 0.000280 0.001968 gc current block 2-­‐way 0.031298 0.0% 47 0.000666 0.000361 0.001640 gc current grant busy 0.028037 0.0% 47 0.000597 0.000156 0.001508 SQL*Net more data from client 0.025819 0.0% 837 0.000031 0.000000 0.002564 CLOSE 0.018999 0.0% 698 0.000027 0.000000 0.001000 12 others 0.061576 0.0% 1,633 0.000038 0.000000 0.001687 -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ TOTAL (22) 932.416074 100.0% 6,477 0.143958 0.000000 4.498316 PARSE calls account for 78.9% of the experience duration. That is never appropriate. @CaryMillsap 80
  81. 81. mrskew -­‐-­‐rc=p10 -­‐-­‐name=parse prd1_ora_9031.trc RANGE {min ≤ e max} DURATION % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ 1. 0.000000 0.000001 0.000000 0.0% 307 0.000000 0.000000 0.000000 2. 0.000001 0.000010 3. 0.000010 0.000100 4. 0.000100 0.001000 0.007992 0.0% 8 0.000999 0.000999 0.000999 5. 0.001000 0.010000 0.033000 0.0% 33 0.001000 0.001000 0.001000 6. 0.010000 0.100000 7. 0.100000 1.000000 8. 1.000000 10.000000 735.385205 100.0% 350 2.101101 1.333797 4.498316 9. 10.000000 100.000000 10. 100.000000 1,000.000000 11. 1,000.000000 +∞ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ TOTAL (11) 735.426197 100.0% 698 1.053619 0.000000 4.498316 That’s a lot of time spent parsing, and these PARSE calls are really expensive. @CaryMillsap 81
  82. 82. mrskew -­‐-­‐name=parse -­‐-­‐group=$sqlid -­‐-­‐gl=SQLID -­‐-­‐top=10 -­‐-­‐sort=4nd prd1_ora_9031.trc SQLID DURATION % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ gkbss8w49204k 4.176363 0.6% 349 0.011967 0.000000 4.135371 66kf30526wrgy 3.153521 0.4% 1 3.153521 3.153521 3.153521 3r3dhkb0z824v 2.911558 0.4% 1 2.911558 2.911558 2.911558 3tzra8a2a7pny 1.605757 0.2% 1 1.605757 1.605757 1.605757 2hycpfzdzsu98 3.155520 0.4% 1 3.155520 3.155520 3.155520 6ppu3s1jszy3a 2.208665 0.3% 1 2.208665 2.208665 2.208665 66vkb784j9rcu 1.901711 0.3% 1 1.901711 1.901711 1.901711 5wamvs45j6nh4 1.492773 0.2% 1 1.492773 1.492773 1.492773 dj1buvhxg7h19 1.499772 0.2% 1 1.499772 1.499772 1.499772 41yrts4g94ghn 1.628753 0.2% 1 1.628753 1.628753 1.628753 340 others 711.691804 96.8% 340 2.093211 1.333797 4.498316 -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ TOTAL (350) 735.426197 100.0% 698 1.053619 0.000000 4.498316 One statement was parsed 349 times; at least 348 of those are unnecessary.* There are 350 distinct SQL statements executed by this report. ...Which is funny, because you know this report, and you don’t remember there being that many. *Actually all 349 are unnecessary, because I can see in the trace data that there’s never an EXEC call associated with any of these PARSE calls, but that’s a story for another day. @CaryMillsap 82
  83. 83. mrskew -­‐-­‐rc=ssqlid prd1_ora_9031.trc SSQLID DISTINCT-­‐TEXTS % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐ -­‐-­‐-­‐ -­‐-­‐-­‐ 4151812497 70 20.0% 70 1 1 1 3642320257 70 20.0% 70 1 1 1 2047770123 70 20.0% 70 1 1 1 1928547239 70 20.0% 70 1 1 1 1138917066 69 19.7% 69 1 1 1 3957414185 1 0.3% 349 0 0 1 -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐ -­‐-­‐-­‐ -­‐-­‐-­‐ TOTAL (6) 350 100.0% 698 1 0 1 For the first 5 “shared SQL id” values shown here, there are ~70 distinct statements that could have been sharable. You should be able to reduce the parse call count from 698 to 6, by writing sharable SQL statements, and pulling PARSE calls out of loops. @CaryMillsap 83
  84. 84. mrskew -­‐-­‐top=10 prd1_ora_9031.trc CALL-­‐NAME DURATION % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ PARSE 735.426197 78.9% 698 1.053619 0.000000 4.498316 SQL*Net message from client 104.762229 11.2% 1,378 0.076025 0.000391 3.554818 FETCH 91.800028 9.8% 680 0.135000 0.000000 0.506923 db file sequential read 0.104670 0.0% 14 0.007476 0.001067 0.016408 EXEC 0.083988 0.0% 349 0.000241 0.000000 0.002000 gc cr block 2-­‐way 0.073233 0.0% 96 0.000763 0.000280 0.001968 gc current block 2-­‐way 0.031298 0.0% 47 0.000666 0.000361 0.001640 gc current grant busy 0.028037 0.0% 47 0.000597 0.000156 0.001508 SQL*Net more data from client 0.025819 0.0% 837 0.000031 0.000000 0.002564 CLOSE 0.018999 0.0% 698 0.000027 0.000000 0.001000 12 others 0.061576 0.0% 1,633 0.000038 0.000000 0.001687 -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ TOTAL (22) 932.416074 100.0% 6,477 0.143958 0.000000 4.498316 Before your boss will let you “fix” this code, you have to predict the benefit. Reducing the parse count from 698 to 6 should reduce parsing duration from ~735 to ~7, a savings of about 730 s. Response time should improve from ~932 s to ~200 s, just from eliminating the PARSE calls only. @CaryMillsap 84
  85. 85. @CaryMillsap You might have known that you should “use bind variables,” but you couldn’t have quantified the R impact on this experience without this trace file. 85 MeTHOD R OPTIMIZE ANYTHING
  86. 86. BASELINE: for each invoice number { cursor = parse(“select ...where invoice_number = ” . number); exec(cursor); loop over the result set to fetch all the rows; } @CaryMillsap 86 BAD This is horrific: • Uses too much CPU for PARSE calls • Serialization on library cache and shared pool latches • Consumes too much memory in the library cache • May execute too many network round-­‐trips
  87. 87. BASELINE: @CaryMillsap BAD for each invoice number { cursor = parse(“select ...where invoice_number = (” . number . “)”); exec(cursor); loop over the result set to fetch all the rows; } FIX 1 “Hey, let’s use bind variables”: for each invoice number { cursor = parse(“select ...where invoice_number = :a1)”); exec(cursor, number); loop over the result set to fetch all the rows; } 87 STILL BAD A little better, but still really awful: • Uses too much CPU for PARSE calls • Serialization on library cache latches • Maybe, too many network round-­‐trips
  88. 88. FIX 1 “Hey, let’s use bind variables”: @CaryMillsap STILL BAD for each invoice number { cursor = parse(“select ...where invoice_number = :a1)”); exec(cursor, number); loop over the result set to fetch all the rows; } FIX 2: cursor = parse(“select ...where invoice_number = :a1)”); for each invoice number { exec(cursor, number); loop over the result set to fetch all the rows; } 88 BETTER Better (only 1 parse call now!), but still lots of network round-­‐trips.
  89. 89. FIX 2: @CaryMillsap BETTER cursor = parse(“select ...where invoice_number = :a1)”); for each invoice number { exec(cursor, number); loop over the result set to fetch all the rows; } FIX 3: cursor = parse(“ select ...where invoice_number in (select invoice number from wherever your for each was getting them) ”); exec(cursor); loop over the result set to fetch all the rows; 89 Now, only 1 PARSE call, and the minimum possible number of network round-­‐trips.* *Unless there’s a way to return fewer rows. BETTER YET
  90. 90. @CaryMillsap And so on... 90
  91. 91. @CaryMillsap Bad SQL Bad PL/SQL Slow network Missing indexes Parsing in a loop Hot block problems Not enough memory Disk latency problems Row locking problems Row-­‐at-­‐a-­‐time processing Bad data structure choice Hardware misconfigurations Too much load on the system OS parameters set inadequately Oracle parameters set inadequately SQL returns more rows than it should Database buffer cache hot/cold problems Oracle query optimizer choosing bad plans Reports run with poorly limiting parameter values Inefficient code between database calls in the application 91 A trace file shows you where your time has gone. Performance problems cannot hide from that.
  92. 92. @CaryMillsap There are only two possible root causes for any response time problem: ❶ Call count is too big. ❷ Latency is too big.* *Probably because someone else’s call counts are too big. 92 #ProTip
  93. 93. mrskew -­‐-­‐top=10 prd1_ora_9031.trc CALL-­‐NAME DURATION % CALLS MEAN MIN MAX -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ PARSE 735.426197 78.9% 698 1.053619 0.000000 4.498316 SQL*Net message from client 104.762229 11.2% 1,378 0.076025 0.000391 3.554818 FETCH 91.800028 9.8% 680 0.135000 0.000000 0.506923 db file sequential read 0.104670 0.0% 14 0.007476 0.001067 0.016408 EXEC 0.083988 0.0% 349 0.000241 0.000000 0.002000 gc cr block 2-­‐way 0.073233 0.0% 96 0.000763 0.000280 0.001968 gc current block 2-­‐way 0.031298 0.0% 47 0.000666 0.000361 0.001640 gc current grant busy 0.028037 0.0% 47 0.000597 0.000156 0.001508 SQL*Net more data from client 0.025819 0.0% 837 0.000031 0.000000 0.002564 CLOSE 0.018999 0.0% 698 0.000027 0.000000 0.001000 12 others 0.061576 0.0% 1,633 0.000038 0.000000 0.001687 -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐ TOTAL (22) 932.416074 100.0% 6,477 0.143958 0.000000 4.498316 See how there are only two ways to reduce a DURATION? You have the CALLS column, and the MEAN column. Profiles like this make it easy to see how anything you do to make something go faster must translate to a manipulation of either CALLS or MEAN. @CaryMillsap 93
  94. 94. @CaryMillsap With a good trace file, you can predict the response time impact of a proposed change.* *This is nearly impossible to do with systemwide aggregated statistics. 94 #ProTip
  95. 95. It just takes practice. @CaryMillsap 95
  96. 96. @CaryMillsap Conclusion 96
  97. 97. @CaryMillsap Your code does stuff. Including some stuff inside Oracle. The time this stuff takes is your user’s response time. You can see exactly what it is. It’s not that hard. 97
  98. 98. @CaryMillsap References 98
  99. 99. @CaryMillsap Robyn Sands, et al. 2010. Expert Oracle Practices. Apress Detailed information about instrumenting your Oracle application code. Cary Millsap. 2011. Mastering Oracle Trace Data. Method R Corporation Textbook for 1-­‐day course that teaches you how to master Oracle trace data. Ron Crisco, et al. 2011. Expert PL/SQL Practices. Apress Detailed information about instrumenting your Oracle application code. Cary Millsap, Jeff Holt. 2003. Optimizing Oracle Performance. O’Reilly Detailed information about Oracle trace data and what to do with it. 99
  100. 100. QA method-­‐r.com www.enkitec.com method-­‐r.com/facebook facebook.com/enkitec @MethodR @Enkitec cary.millsap@method-­‐r.com cary.millsap@enkitec.com @CaryMillsap 100

×