2. Who is Kyle Hailey
1990 Oracle
–
–
–
–
–
90 support
92 Ported v6
93 France
95 Benchmarking
98 ST Real World Performance
2000 Dot.Com
2001 Quest
2002 Oracle OEM 10g
Success!
First successful OEM design
3. Who is Kyle Hailey
1990 Oracle
–
–
–
–
–
90 support
92 Ported v6
93 France
95 Benchmarking
98 ST Real World Performance
2000 Dot.Com
2001 Quest
2002 Oracle OEM 10g
2005 Embarcadero
– DB Optimizer
4. Who is Kyle Hailey
•
1990 Oracle
•
•
•
•
•
90 support
92 Ported v6
93 France
95 Benchmarking
98 ST Real World Performance
2000 Dot.Com
2001 Quest
2002 Oracle OEM 10g
2005 Embarcadero
DB Optimizer
Delphix
When not being a Geek
- Have a little 4 year old boy who takes up all my time
27. Clearer
Damage on every flight
below 65
12
8
4
55
60
65
70
75
80
1. Y-Axis amount of damage (not number of damage)
2. Include successes
3. Mark Differences
4. Normalize same temp
No damage on every
flight above 75
30. Difficult
NASA Engineers Fail
Congressional Investigators Fail
Data Visualization is Difficult
But …
Lack of Clarity can be devastating
31. Solutions
• Clear Identification
•
Know how to identify problems and issues
• Access to details
•
Provide solutions and/or information to address the issues
• Graphics
•
Easy understanding, effective communication and discussion
32. First Step: Graphics
“The humans … are exceptionally good
at parsing visual information,motion
especially when that information is
coded by color and/or
.”
Knowledge representation in cognitive science. Westbury, C. & Wilensky, U. (1998)
33. Why Use Graphics
You can't imagine how many times I was told that nobody wanted or
would use graphics …
-- Jef Raskin, the father of the Macintosh
Infocus – (overhead projectors) sited a
study that humans can parse
graphical information 400,000 times
faster than textual data
40. Tuning the Database
Complex
What is a Averages life look
day in the
like for a DBA who has
performance issues?
I
y
x
Average
Standard Deviation
Linear Regression
9
3.31
1.33
x
7.5
2.03
Anscombe's Quartet
II
III
y
x
y
x
9
7.5
9
7.5
3.31
2.03
3.31
2.03
1.33
1.33
IV
y
9
3.31
1.33
7.5
2.03
42. Imagine Trying to Drive your
Car
Would you want your dashboard to look like :
And is updated once and hour
Or would you like it to
look …
43. How Can We Open the Black Box?
LOAD
Max CPU
(yard stick)
Top Activity
SQL
Sessions
44. How Can We Open the Black Box?
OEM
ASHMON/SASH
DB Optimizer
• Powerful - Identifies issues quickly and indepth
• Interactive - Allows exploring the data
• Easy - Understandable by everyone, DBA, Dev and Managers
!
47. In this Session
1. AAS
–
–
Single Metric
DB Performance
1. Yardstick
–
–
CPU Count
Compare to AAS
1. AAS components
–
–
–
10/18/13
CPU
Waits
Time series
47
48. Database Performance
How quick can you find
Bottleneck in DB
If DB is idle
Current DB Load
what is DB Load ?
What do you use?
Statspack/AWR
V$active_session_history
Alerts
what the *!####!
*!*? is the
database
doing ?!
what do you alert on ?
10/18/13
48
51. Statspack Sections (10g)
Instance description
Host hardware
Snapshot headline
Cache information
Load profile
Instance efficiency
Shared pool stats
Top timed events
Host CPU load
Host / Instance CPU warning
VM activity
Memory usage
Time model stats
RAC statistics
All wait events
Background wait events
Event histogram
SQL x 9
Instance activity
Log switches
Session details x 3
Tablespace I/O
File I/O
File I/O histogram
Buffer pool
Instance Recovery
Buffer pool advisory
Buffer busy waits
Various PGA
summaries
PGA histogram
PGA advisory
PGA allocation
summary
PGA allocation top
N
Enqueue (lock)
activity
Undo stats x 2
Latch activity
Latch miss details
Latch parent and child
Mutex
Segment stats x 8
Dictionary cache
Library cache
RAC (GES)
RAC (CR and CUR
served)
RAC (cache xfer x 2)
RAC (Remastering)
Streams x 7
Shared pool advisor
Java pool advisor
SGA resizing
SGA target advisor
SGA summary
SGA detail
SQL memory summary
Resource limits
Parameters
52. Statspack Method
Who/When
Summary
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
% for
Load profile goodTotal
Event
Waits having a (s) Call Time
Time feel for the
---------------------- ------------ application and
----------- --------buffer busy waits
2,748
250
78.72
comparing two 10.16
CPU time
32
periods for changes
free buffer waits
1,588
15
4.63
Efficiency 8
ratios 2.51
write complete waits
10
log buffer space
306 misleading carry 1.51
5
---------------------------------------- 6
over from version
days
Waits
Big Picture
53. Statspack Method
v$system_event - wait events
v$sysstat – cpu stats
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
% Total
Event
Waits
Time (s) Call Time
---------------------- ------------ ----------- --------buffer busy waits
2,748
250
78.72
CPU time
32
10.16
free buffer waits
1,588
15
4.63
write complete waits
10
8
2.51
log buffer space
306
5
1.51
----------------------------------------
Old method:
compare wait time to CPU time
54. Current Method
•
•
•
1000 lines of data / 30 pages
What do you look at ?
Top 5 Timed Events?
188 minutes CPU
Elapsed 15 minutes
48 Processors or
74% idle
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
% Total
Event
Waits
Time (s) Ela Time
------------------------ ------------ ----------- -------CPU time
11,330
81.95
log file sync
163,725
1,012
7.32
enqueue
4,486
615
4.45
latch free
28,786
236
1.71
db file sequential read
1,420,355
149
1.08
----------------------------------------Jonathan Lewis - http://www.jlcomp.demon.co.uk/statspack_01.html
Analysis requires
•elapsed time
•CPU core count
55. Average Active Sessions (AAS)
DB Time = wait time + CPU time
AAS = DB Time/Elapsed Time
AAS like run queue on UNIX
60. Statspack AAS
• Elapsed Time
STATSPACK report for
STATSPACK report for
LABSF03 1420044432
1 10.1.0.2.0 NO labsfr
LABSF03 1420044432 labsf03
labsf03
1 10.1.0.2.0 NO labsfr
Snap Id
Snap Time
Sessions Curs/Sess
Snap Id
Snap Time
Sessions Curs/Sess
Begin Snap:
1 03-Apr-06 12:34:06
18
5.6
Begin Snap:
1 03-Apr-06 12:34:06
18
5.6
End Snap:
2 03-Apr-06 12:34:36
18
4.8
End Snap:
2 03-Apr-06 12:34:36
18
4.8
Elapsed:
1.00 (mins)
Elapsed:
1.00 (mins)
• Look at Top 5 Timed Events
Top 5 Timed Events
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~
Event
Event
buffer busy waits
buffer busy waits
CPU time
CPU time
free buffer waits
free buffer waits
write complete waits
write complete waits
log buffer space
log buffer space
10/18/13
Waits
Waits Time (s)
Time (s)
2,748
250
2,748
250
32
32
1,588
15
1,588
15
10
8
10
8
306
5
306
5
% Total
% Total
Call Time
Call Time
78.72
78.72
10.16
10.16
4.63
4.63
2.51
2.51
1.51
1.51
60
61. Statspack AAS
• DBTIME= CPU + WAITS
– CPU = 32
– WAITS = 250+15+8+5 = 278 secs
DBTIME=320
• Elapsed Time = 60 secs
•
320 secs / 60 secs
Top 5 Timed Events
Top 5 Timed Events
Event
Time (s)
Event
Time (s)
--------------------------------- --------buffer busy waits
250
buffer busy waits
250
CPU time
32
CPU time
32
free buffer waits
15
free buffer waits
15
write complete waits
8
write complete waits
8
log buffer space
5
log buffer space
5
AAS = 5.1
CPU core count = 2
61
62. AAS Formulas
Use CPU count as yardstick:
AAS < 1
Database is not blocked
AAS ~= 0
Database basically idle
Problems are in the APP not DB
Ideal world –
one database
solution
track CPU at OS
AAS < # of CPUs
CPU available
Are any single sessions 100% active?
•
AAS > # of CPUs
Could have performance problems
AAS >> # of CPUS
There is a bottleneck
AAS > 1
still want to know
if a single user is
100% active
64. What’s the DB Doing?!
It’s 2am … your manager calls
Whip out the stethoscope:
AAS
what the *!####!*!
*? is the database
doing ?!
10/18/13
Copyright 2006 Kyle Hailey
64
68. Latch Free
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
% Total
Event
Waits
Time (s) Ela Time
-------------------------------------------- ------------ ----------- -------latch free
9,652
760
66.10
CPU time
248
21.62
PL/SQL lock timer
41
123
10.72
SQL*Net message from dblink
681
14
1.22
log file parallel write
128
1
.13
-------------------------------------------------------------
What Latch? There are 100s
Latch Sleep breakdown for DB: CDB Instance: cdb Snaps: 3 -4
-> ordered by misses desc
Latch Name
Requests
Misses
Sleeps
-------------------------- -------------- ----------- ----------cache buffers chains
8,448,787
649,484
6,930
library cache pin
8,405,896
82,915
1,427
library cache
shared pool
8,435,488
55,645
58,626
7
Sleeps 1->4
-----------0/0/0/0/0
81537/1330/4
7/1/0
1,294 54375/1247/2
2/1/0
1 6/1/0/0/0
69. Row Locks 10g+
op 5 Timed Events
Avg %Total
~~~~~~~~~~~~~~~~~~
wait
Call
Event
Waits
Time (s)
(ms)
Time
----------------------------------------- ------------ ----------- ------ -----enq: TX - row lock contention
59
160
2714
41.8
PL/SQL lock timer
4
117 29291
30.6
CPU time
28
7.2
buffer busy waits
1,217
18
15
4.7
log file parallel write
422
11
27
3.0
Who is waiting
Who is blocking
What is the SQL
What is the row?
70. Buffer Busy Wait
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
% Total
Event
Waits
Time (s) Call Time
-------------------------------------------- ------------ ----------- --------buffer busy waits
2,748
250
78.72
CPU time
32
10.16
free buffer waits
1,588
15
4.63
write complete waits
10
8
2.51
log buffer space
306
5
1.51
------------------------------------------------------------
Buffer Busy Wait – trying to modify a block
Who blocks? What object? what is the SQL?
Buffer wait Statistics DB/Inst: DB4/db4 Snaps: 2-3
Class
Waits Wait Time (s) Avg Time (ms)
------------------ ----- ------------- ------------file header block
264
203
769
data block
6,070
162
27
undo header
355
0
1
Statspack fails for analysis
73. Dilema: AAS indicates problem but not
solutions
Problem in
•Statspack
•AWR
•OEM performance page
Can’t analyze
•Locks
•Buffer Busy Waits
•Latches
Also can’t see when demand for CPU is greater than
available CPU
75. Example
•
Look at Statspack:
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~
% Total
Event
Waits
Time (s) Call Time
--------------------------- ----------- --------buffer busy waits
2,748
250
78.72
CPU time
32
10.16
free buffer waits
1,588
15
4.63
write complete waits
10
8
2.51
log buffer space
306
5
1.51
-------------------------------
76. What do we do?
•
•
How do we solve “buffer busy wait”
Need:
•
•
•
•
SQL
Sessions
Objects
Wait details
– Type of buffer busy wait
– File and block numbers
•
How do we get that information?
– Not from Statspack or AWR
– But from v$session
• v$session_wait prior to 10g
77. V$session
•
Shows who is waiting
– For example lists who is waiting on “buffer busy waits”
– Data only exists while waits are happening
– Data includes
• Type of buffer busy wait
• File and block involved in buffer busy wait
•
•
Problem: Once waits are over, data is gone
Solution: Sample data all the time
78. Sample Query
select
nvl(s.username,s.program) username,
s.sid sid,
s.serial# serial,
s.sql_hash_value sql_hash_value, SQL identifier
substr(decode(w.wait_time,
Waiting or on CPU?
0, w.event,
'ON CPU'),1,15) event ,
w.p1 p1,
w.p2 p2,
w.p3 p3
from
v$session
s,
v$session_wait w
where w.sid=s.sid
and s.status='ACTIVE'
and s.type='USER';
This query works since v7
80. Buffer busy wait type
SQL> Select count(*), p3
from v$ash
where event = 'buffer busy waits'
group by p3;
COUNT(*)
P3
---------- ---3423
1
P3 = block type
81. Buffer busy wait type
N CLASS
Block types come from
select rownum n,
class
from v$waitstat;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
data block
sort block
save undo block
segment header
save undo header
free list
extent map
1st level bmb
2nd level bmb
3rd level bmb
bitmap block
bitmap index block
file header block
unused
system undo header
system undo block
undo header
82. File and Block #s
select count(*), p1 filen, p2 blockn
from v$ash
where event='buffer busy waits'
group by p1, p2, hash_value;
COUNT(*)
FILEN
BLOCKN
---------- -------- -------1
11
90644
2
11
90651
3
11
98233
1
11
104767
3
11
113291
1
11
119842
1
11
119856
3
11
121632
1
11
126334
Pick a File and Block to find object
83. Find Object
column segment_name format a30
select
COUNT(*)
FILEN
BLOCKN
---------- -------- -------1
11
126334
owner,
segment_name,
segment_type,
block_id, blocks+block_id
from dba_extents
where file_id = 11
and 126334 between block_id AND block_id + blocks-1;
OWNER
SEGMENT_NAME
SEGMENT_TY
BLOCK_ID BLOCKS+BLOCK_ID
---------- ------------------ ---------- ---------- --------------SYSTEM
TOTO1
TABLE
125201
127249
Data block ( not header block )
126334
84. What SQL ?
SQL> select count(*), sql_hash_value
2 from v$ash
3 where event='buffer busy waits'
4 group by hash_value;
COUNT(*)
---------3423
SQL_HASH_VALUE
-------------558666863
All the same SQL
SQL_HASH_VALUE= 558666863
In version 10g+ use SQL_ID
85. SQL Statement ?
select sql_text
from v$sqltext
where hash_value=558666863;
SQL_TEXT
----------------------------------------------------INSERT into toto1 values (:b1,lpad('a',1000,'a'))
• Insert statement
• Problem on a data block on
• table toto1
Solution: Freelists or ASSM
86. Sampling Summar y
• Sampling v$session
– finds the root of problems
– otherwise impossible to diagnose
• Sample Data from v$session is called
– Active Session History (ASH)
– v$active_session_history
Diagnosis is still complex
But
done automatically by ADDM
87. OEM & ADDM Example
Four Tuning Paths
Database Home Page
Performance Page
Drilldowns
SQL
Session
ADDM Page
Solutions
114. In Review
•
By using
– Free Lists
– Large Redo Log files
•
Job Processing went from
– Over 14 minutes to
– Just over 1 minute
•
Thanks to ADDM and OEM !
121. Primary Fields of ASH
Activity : Who, What, When, How
When
SAMPLE_TIME
Who
SESSION_ID
Session
SESSION_STATE
State
EVENT
Wait
SQL_ID
SQL
Time
How
What
131. The Power
DB Home
ASH gives
AAS
Performance
Based on TIME
AAS=db
time/elapsed time
v$system_event
v$sysstat
Top Activity
AAS=count active
users /samples
Based on
ASH
Copyright 2006 Kyle Hailey
133. 2. OEM 10g : Top Activity
•Top Activity
•Based on ASH
•Enables Drilldowns
•Top SQL
•Top Session
•Drill into a session
• Stats
• Raw waits
• Open cursors
• General info
•Drill into a SQL
• Stats and text
• Users executing
• Explain plan
• Tuning options
137. AAS from ASH
1.
ASHRPT
–
Based entirely on v$active_session_history
–
–
@?/rdbms/admin/ashrpt.sql
Exec ASH_REPORT_TEXT/HTML
select * from table
(dbms_workload_repository.ash_report_text(
(select dbid from v$database),
1,
sysdate – 1/24,
sysdate )) ;
10/18/13
137
138. ASHRPT
ASH Report For TESTDB/testdb
ASH Report For TESTDB/testdb
DB Name
DB Id
Instance
DB Name
DB Id
Instance
TESTDB
TESTDB
CPUs
CPUs
Inst Num Release
Inst Num Release
2371570538 testdb
2371570538 testdb
SGA Size
Buffer Cache
SGA Size
Buffer Cache
RAC Host
RAC Host
11 10.2.0.1.0 NO sdbe604a
10.2.0.1.0 NO sdbe604a
Shared Pool
ASH Buffer Size
Shared Pool
ASH Buffer Size
22
1,000M (100%)
468M (46.8%)
112M (11.2%)
4.0M (0.4%)
1,000M (100%)
468M (46.8%)
112M (11.2%)
4.0M (0.4%)
Analysis Begin Time:
21-Apr-06 12:00:01
Analysis Begin Time:
21-Apr-06 12:00:01
Analysis End Time:
21-Apr-06 12:05:01
Analysis End Time:
21-Apr-06 12:05:01
Elapsed Time:
5.0 (mins)
Elapsed Time:
5.0 (mins)
Sample Count:
3,716
Sample Count:
3,716
Average Active Sessions:
12.39
Average Active Sessions:
12.39
Avg. Active Session per CPU:
6.19
Avg. Active Session per CPU:
6.19
Report Target:
None specified
Report Target:
None specified
Top User Events
DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)
Top User Events
DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)
Avg Active
Avg Active
Event
Event Class
%% Activity
Event
Event Class
Activity Sessions
Sessions
CPU ++ Wait for CPU
CPU
Wait for CPU
enq: TX -- row lock contention
enq: TX
row lock contention
buffer busy waits
buffer busy waits
latch: cache buffers chains
latch: cache buffers chains
CPU
CPU
Application
Application
Concurrency
Concurrency
Concurrency
Concurrency
67.98
67.98
23.98
23.98
4.66
4.66
2.26
2.26
8.42
8.42
2.97
2.97
0.58
0.58
0.28
0.28
139. ASH RPT
1) General info
1) General info
2) Top User Events ***
2) Top User Events ***
3) Top Background Events
3) Top Background Events
4) Top Event P1/P2/P3 Values
4) Top Event P1/P2/P3 Values
5) Top Service/Module
5) Top Service/Module
6) Top Client IDs
6) Top Client IDs
7) Top SQL Command Types
7) Top SQL Command Types
8) Top SQL Statements ***
8) Top SQL Statements ***
9) Top SQL using literals
9) Top SQL using literals
10) Top Sessions ***
10) Top Sessions ***
11) Top Blocking Sessions
11) Top Blocking Sessions
12) Top Sessions running PQs
12) Top Sessions running PQs
13) Top DB Objects
13) Top DB Objects
14) Top DB Files
14) Top DB Files
15) Top Latches
15) Top Latches
16) Activity Over Time ***
16) Activity Over Time ***
140. ASHRPT over Time
• Waits over Time
– Not in AAS
• Difficult but better
than nothing
Compare to …
141. Custom Scripts
• Hate Graphics ?
– Query v$active_session_history directly
– Join to dba_hist_active_sess_history for week of data
• act.sql
– Like top 5 timed events
• Aveact.sql
– Charts with text AAS by hour (15 minute, minute , etc)
• Aveactn.sql
– Ditto, with top 2 wait events per bucket
• Following Scripts Available on
http://ashmasters.com
10/18/13
141
142. Custom Scripts
@act
@act
Analysis Begin Time :
Analysis Begin Time : 2007-07-24 11:04:48
2007-07-24 11:04:48
Analysis End
Analysis End Time :
Time : 2007-07-24 11:19:45
2007-07-24 11:19:45
Start time, mins ago:
15
Start time, mins ago:
15
Request Duration
:
15
Request Duration
:
15
Collections
:
528
Collections
:
528
Data Values
:
3327
Data Values
:
3327
Elapsed Time: 15 mins
Elapsed Time: 15 mins
WAIT_EVENT
WAIT_EVENT
latch free
latch free
log buffer space
log buffer space
buffer busy waits
buffer busy waits
db file scattered read
db file scattered read
library cache pin
library cache pin
log file sync
log file sync
ON CPU
ON CPU
enqueue
enqueue
db file sequential read
db file sequential read
sum
sum
CNT
CNT
10
10
13
13
14
14
15
15
78
78
213
213
726
726
855
855
1399
1399
% Active Ave_Act_Sess
% Active Ave_Act_Sess
.3
.02
.3
.02
.39
.02
.39
.02
.42
.03
.42
.03
.45
.03
.45
.03
2.34
.15
2.34
.15
6.40
.40
6.40
.40
21.82
1.38
21.82
1.38
25.70
1.62
25.70
1.62
42.05
2.65
42.05
2.65
6.30
6.30
147. Before Tuning Oracle, Tune the Machine
Make sure the machine is healthy before tuning Oracle
CPU => use run queue, < 2 * #CPU
Memory => page out
VMSTAT
148. Summary
1.Machine - vmstat
–
Memory, CPU (we can see IO response in Oracle)
1.Database - AAS
–
Use wait interface and graphics
–
Identify machine, application, database or SQL
1.SQL - VST
–
Indexes, stats, execution path
–
Visual SQL Tuning
149.
150. Bibliography
Refactoring SQL Applications - Stephane Faroult
Troubleshooting Oracle Performance - Christian Antognini
SQL Tuning - Dan Tow
Oracle Wait Interface - Kirtikumar Deshpande, K.
Gopalakrishnan
151. Books
The Visual Display of
Design
Quantitative Information
– Eduard Tufte
For the Space Shuttle disaster, see
http://williamwolff.org/wp-content/uploads/2013/01/tufte-challenger-1997.pdf
http://www.onlineethics.org/cms/12709.aspx
153. ASSM ?
select FREELISTS, TABLESPACE_NAME
from dba_tables
where table_name='TOTO1' and owner='SCOTT';
FREELISTS
---------1
TABLESPACE_NAME
-----------------------------USERS
Table only has 1 freelist
The table is in tablespace Users
Is it an ASSM tablespace ?
Copyright 2006 Kyle Hailey
154. Sampling Output
select tablespace_name, SEGMENT_SPACE_MANAGEMENT
from dba_tablespaces;
TABLESPACE_NAME
-----------------------------SYSTEM
UNDOTBS1
TEMP
USERS
TOOLS
SEGMEN
-----MANUAL
MANUAL
MANUAL
MANUAL
AUTO
USERS tablespace is not in ASSM mode
Copyright 2006 Kyle Hailey
155. Let’s add Freelists
How many freelists?
select count(*), sid, serial
from v$ash
where event='buffer busy waits‘
group by sid, serial;
COUNT(*)
SID SERIAL
---------- ----- -------7
12
79
4
14
99
4
19
770
8
20
176
Max Concurrency is 4
Freelists >= 4
Copyright 2006 Kyle Hailey
Notes de l'éditeur
{"115":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","60":"Statspack is probably the most reliable source of performance information.\nThe statspack report, ?/rdbms/admin/spreport.sql , generates over 1000 lines of information, but the first and possible only place to go in the report is “Top 5 Timed Events”.\nIn “Top 5 Timed Events” we can determine if the database has any performance issues. If it does have performance issues, then we can find out if it’s CPU or a wait. If it’s a wait we can tune that particular wait.\n","49":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","16":"The O-rings of the solid rocket boosers were not designed to erode.\nErosion was a clue that something was wrong. \nErosion was not something from which safety could be inferred\n- Richard Feynman\n","33":"What allows Oracle, all the dials and knobs, to be the fastest, most robust database on the market also make it one of the most complicated on the market. For those few users that know and love the 1000s of performance options in Oracle, access to those performance options through sql and C code is the defacto standard. For seasoned users it is often difficult to understand why anyone would want to be limited by a GUI which can often be slower, and almost always lacking the latest commands allowed through interfaces such as SQL and C. \nOn the OEM side, when a new tuning feature in the kernel is externalized in OEM (Oracle Enterprise Manager) it is often difficult to see the forest for the tree, or where and how that new feature fits into the grand scheme of things. \n","127":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","116":"If happens a lot or for long … we’ll catch it, guaranteed\n","122":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","56":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","45":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","117":"To execute a query , some CPU will be used but we also might spend time waiting for IO or on waiting for concurrency resources like latches\n","2":"In Oracle Support I learned more faster than I think I could have anywhere.\nPorting gave me my first appreciation for the shareable nature of Oracle code and also a bit of disbelief that it worked as well as it did.\nOracle France gave me an opportunity to concentrate on the Oracle kernel.\nAt Oracle France I had 3 amazing experiences. First was being sent to the Europecar site where I first met a couple of the people who would later become the founding members of the Oaktable, James Morle and Anjo Kolk. The Europecar site introduced me to a fellow name Roger Sanders who first showed me the wait interface before anyone knew what it was. Roger not only used it but read it directly from shared memory without using SQL.\nSoon after Europecar I began to be sent often to benchmarks at Digital Europe. These benchmarks were some of my favorite work at Oracle. The benchmarks usually consisted of installing some unknown Oracle customer application and then having a few days to make it run as fast as possible. I first started using TCL/TK and direct shared memory access (DMA) at Digital Europe and got solid hands on tuning experience testing things like striping redo and proving it was faster long before people gave up arguing that this was bad from a theoretical point of view.\nFinally in France, my boss, Jean Yves Caleca was by far the best boss I ever had, but on top of that he was wonderful at exploring the depths of Oracle and explaining it to others, teaching me much about the internals of block structure, UNDO, REDO and Freelsits.\nI came back from France wanting to do performance work and especially graphical monitoring. The kernel performance group had limited scope in that domain, so I left for a dot com where I had my first run as the sole DBA for everything, backup, recovery, performance, installation and administration. I was called away by Quest who had my favorite performance tool Spotlight. It turns out thought that scope for expanding Spotlight was limited so I jumped at the chance in 2002 to restructure Oracle OEM. The work at OEM I’m proud of but still want to do much more to make performance tuning faster, easier and more graphical.\n","118":"To execute a query , some CPU will be used but we also might spend time waiting for IO or on waiting for concurrency resources like latches\nActivity can thus further be broken down into the type of activity: CPU, IO or WAIT\n","74":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","3":"In Oracle Support I learned more faster than I think I could have anywhere.\nPorting gave me my first appreciation for the shareable nature of Oracle code and also a bit of disbelief that it worked as well as it did.\nOracle France gave me an opportunity to concentrate on the Oracle kernel.\nAt Oracle France I had 3 amazing experiences. First was being sent to the Europecar site where I first met a couple of the people who would later become the founding members of the Oaktable, James Morle and Anjo Kolk. The Europecar site introduced me to a fellow name Roger Sanders who first showed me the wait interface before anyone knew what it was. Roger not only used it but read it directly from shared memory without using SQL.\nSoon after Europecar I began to be sent often to benchmarks at Digital Europe. These benchmarks were some of my favorite work at Oracle. The benchmarks usually consisted of installing some unknown Oracle customer application and then having a few days to make it run as fast as possible. I first started using TCL/TK and direct shared memory access (DMA) at Digital Europe and got solid hands on tuning experience testing things like striping redo and proving it was faster long before people gave up arguing that this was bad from a theoretical point of view.\nFinally in France, my boss, Jean Yves Caleca was by far the best boss I ever had, but on top of that he was wonderful at exploring the depths of Oracle and explaining it to others, teaching me much about the internals of block structure, UNDO, REDO and Freelsits.\nI came back from France wanting to do performance work and especially graphical monitoring. The kernel performance group had limited scope in that domain, so I left for a dot com where I had my first run as the sole DBA for everything, backup, recovery, performance, installation and administration. I was called away by Quest who had my favorite performance tool Spotlight. It turns out thought that scope for expanding Spotlight was limited so I jumped at the chance in 2002 to restructure Oracle OEM. The work at OEM I’m proud of but still want to do much more to make performance tuning faster, easier and more graphical.\n","119":"To execute a query , some CPU will be used but we also might spend time waiting for IO or on waiting for concurrency resources like latches\nActivity can thus further be broken down into the type of activity: CPU, IO or WAIT\n","15":"The O-rings of the solid rocket boosers were not designed to erode.\nErosion was a clue that something was wrong. \nErosion was not something from which safety could be inferred\n- Richard Feynman\n","4":"In Oracle Support I learned more faster than I think I could have anywhere.\nPorting gave me my first appreciation for the shareable nature of Oracle code and also a bit of disbelief that it worked as well as it did.\nOracle France gave me an opportunity to concentrate on the Oracle kernel.\nAt Oracle France I had 3 amazing experiences. First was being sent to the Europecar site where I first met a couple of the people who would later become the founding members of the Oaktable, James Morle and Anjo Kolk. The Europecar site introduced me to a fellow name Roger Sanders who first showed me the wait interface before anyone knew what it was. Roger not only used it but read it directly from shared memory without using SQL.\nSoon after Europecar I began to be sent often to benchmarks at Digital Europe. These benchmarks were some of my favorite work at Oracle. The benchmarks usually consisted of installing some unknown Oracle customer application and then having a few days to make it run as fast as possible. I first started using TCL/TK and direct shared memory access (DMA) at Digital Europe and got solid hands on tuning experience testing things like striping redo and proving it was faster long before people gave up arguing that this was bad from a theoretical point of view.\nFinally in France, my boss, Jean Yves Caleca was by far the best boss I ever had, but on top of that he was wonderful at exploring the depths of Oracle and explaining it to others, teaching me much about the internals of block structure, UNDO, REDO and Freelsits.\nI came back from France wanting to do performance work and especially graphical monitoring. The kernel performance group had limited scope in that domain, so I left for a dot com where I had my first run as the sole DBA for everything, backup, recovery, performance, installation and administration. I was called away by Quest who had my favorite performance tool Spotlight. It turns out thought that scope for expanding Spotlight was limited so I jumped at the chance in 2002 to restructure Oracle OEM. The work at OEM I’m proud of but still want to do much more to make performance tuning faster, easier and more graphical.\n","120":"WAIT_TIME is time of last wait, > 0 , means on CPU\nTIME_WAITED is the actual wait time\n"}