Evaluate how certain real world database workload behaves on different I/O subsystem, processors and
architecture or the coexistence with other databases is the goal of a Proof of Concept. The need of testing
real production workloads to eliminate uncertainty with help of techniques like Workload Folding, Time
Shifting and Schema Remapping, this talk will produce evidence that exploring Real Application Testing
features in 12c leverage what can be accomplished by a Proof of Concept.
3. We will discuss
Database Replay features on 12c version and how we
can apply stress test and workload scale-up techniques to
ensure a real world proof of concept
SQL Perfomance Analyzer
Actual PoC results
We will not to discuss
Luís Marques - @drune - http://lcmarques.com
4. PoC – What really is?
An huge amount of work to plan and execute (and re-
execute until the end of available time period)
Most work is for DBA if Real Application Testing is used.
Luís Marques - @drune - http://lcmarques.com
5. The (real) way it was…
Hardware arrives
Setup required
Recent Backup
required
Restore your DB into
new hardware
Flashback Guaranteed
Restore PointSetup Application:
- New hardware
- Install and Configure:
- AS, Clients, Network, etc
We are
ready!
2 weeks of infinite fun
and infinite problems
Luís Marques - @drune - http://lcmarques.com
6. The (real) way it was…
Luís Marques - @drune - http://lcmarques.com
Database scheduler Jobs
created on-the-fly
Small reference
in docs
(not even in
restrictions area)
7. The way it should be
Working with the vendor to provide support for Real
Application Testing
Client requested a simple and effective way to test new
platform and software releases
Luís Marques - @drune - http://lcmarques.com
8. The way it should be
Hardware arrives
Setup required
Recent Backup
required
Restore your DB into
new hardware using
START_SCN from Capture
Flashback Guaranteed
Restore Point
DB Replay
DB Capture
Luís Marques - @drune - http://lcmarques.com
10. Before start any capture…
RAT Patches on Production and PoC database (Doc ID
560977.1)
Capture Restrictions: direct path load external files with
SQL*Loader, scheduler jobs, flashback querys, distributed trx, etc..
Workload is important: sysdate use,avoid maintenance windows
or EM activity
No ongoing transactions during capture - Plan you
database shutdown and start capture immediately after startup
Luís Marques - @drune - http://lcmarques.com
12. Workload Capture
Populate DBA_WORKLOAD_CAPTURES / FILTERS
SQL> exec dbms_workload_capture.get_capture_info('DB_REPLAY_DIR');
Save the START_SCN to use with RMAN UNTIL SCN
to restore your database on PoC database.
Luís Marques - @drune - http://lcmarques.com
13. Workload Capture
Export AWR data to compare performance between
Production and PoC database:
SQL> exec dbms_workload_capture.export_awr(2);
Dump of AWR will be used after replay to compare AWR on
PoC database
AWR data
Luís Marques - @drune - http://lcmarques.com
14. Workload Capture – The intelligent stuff
Workload Intelligence – Java programs to analyze
Captured data
Don’t run it on production system (Heavy memory
consumption)
Allow you to know more about the capture by patterns
Create user
wi
Create
Workload
Job
Create
Workload
Model
Identify
Patterns
Generate
Report
Java –classpath … oracle.dbreplay.workload.intelligence.LoadInfo -job wi_job1
-cdir /home/oracle/scripts/rat/db_replay_dir -cstr
jdbc:oracle:thin:@baco:1521:bacodb2 -user wi
Java –classpath … oracle.dbreplay.workload.intelligence.BuildModel -job
wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi
Java –classpath … oracle.dbreplay.workload.intelligence.FindPatterns -job
wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi
Java –classpath … oracle.dbreplay.workload.intelligence.GenerateReport -job
wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi –top 5 –out
intel_report.html
Luís Marques - @drune - http://lcmarques.com
15. How an intelligence report looks…
Pattern is a sequence of transactions that run under “certain order”.
Allow you to identify mostly used “code paths” – Example: create a customer or
process a visa card “code paths”
Patterns are ordered by different columns (DB Time, Nr. Executions and Length)
Pattern id = 2 Create a customer run 2333 times during the capture with a DB
Time 9 seconds (0,78% of total DB Time)
Luís Marques - @drune - http://lcmarques.com
Pattern id = 2
Pattern id =1
17. Before start any replay…
Setup guarantee restore points before any replay
Run java utility “Workload Analyzer” – Check the
quality
of capture (insufficient data, errors, etc)
Run a small test replay (~ 20 min) before full replay -
Avoid potential issues with full replay
Set up your system date to the same time as time of
capture start if application relies on sysdate
Before performance analysis perform a replay analysis
and review divergences
Luís Marques - @drune - http://lcmarques.com
18. Before start replay - sysdate
Set system clock on PoC system to the start of time of
capture
Make sure that NTP daemon is disabled
Make sure that AWR snapshots are still created
automatically after system clock changed
Restart PoC database after change in the clock.
Luís Marques - @drune - http://lcmarques.com
19. Before start replay – The Workload
Analyzer
Java utility provided by Oracle to analyze the quality of
your capture.
$ java -classpath
$ORACLE_HOME/jdbc/lib/ojdbc6.jar:$ORACLE_HOME/rdbms/jlib/dbrparser.jar:$ORACLE_H
OME/rdbms/jlib/dbranalyzer.jar: oracle.dbreplay.workload.checker.CaptureChecker
/home/oracle/scripts/rat/db_replay_dir
jdbc:oracle:thin:@baco.localdomain:1521:prodpay2
Almost all sessions have been captured in-flight.
PL/SQL blocks or functions have 'complicated' logic or
multiple commits in them, they are hard to synchronize and
they behavior might change during replay.
wcr_cap_analysis.html
Luís Marques - @drune - http://lcmarques.com
20. Captured workload: Sequential payment processing by Card type
Goal: Parallelized payment processing by Card Type
Scenario #1
Luís Marques - @drune - http://lcmarques.com
21. Consolidate Replay – Workload folding
Consolidate Database Replay: Slicing an existing captured
workload into subsets by specifying a point in time
Allows scale-in testing replaying workload subsets together
Our Goal: Parallelized payment processing by Card Type
Start of Day
Processing
VISA Payment
Process
MasterCard
Payment
Process
OthersCard
Payment
Process
Workload
TimeWL Subset
#1
WL Subset
#2
WL Subset
#3
Luís Marques - @drune - http://lcmarques.com
22. Consolidate Replay – Workload folding
Workload from 1 day (7pm to 6am) – capture_payment_day
3 subsets of workload derived from 1 capture:
Start of Day
Processing
VISA Payment
Process
MasterCard
Payment
Process
OthersCard
Payment
Process
Workload
Time
Day Finish
Replay this!
Luís Marques - @drune - http://lcmarques.com
23. Consolidate Replay – Workload folding
BEGIN_TIME and END_TIME are offset in seconds from
the start of the capture (tricky)
Incomplete calls are discarded to avoid problematic results
Example (VISA workload subset #1):
• 4500 seconds after start of capture (1 hour and 15 minutes)
• Ends after 8100 seconds from the start of capture (1 hour of duration)
SQL> EXEC DBMS_WORKLOAD_REPLAY.GENERATE_CAPTURE_SUBSET (
INPUT_CAPTURE_DIR => 'DB_REPLAY_DIR',
OUTPUT_CAPTURE_DIR => 'DB_REPLAY_VISA',
NEW_CAPTURE_NAME => 'CardVisa_wkld’,
BEGIN_TIME => 4500,
BEGIN_INCLUDE_INCOMPLETE=> FALSE,
END_TIME => 8100,
END_INCLUDE_INCOMPLETE=> FALSE,
PARALLEL_LEVEL=>1)
Luís Marques - @drune - http://lcmarques.com
24. Consolidate Replay – Workload folding
Preprocess capture from all directories:
EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_VISA');
EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_MCARD');
EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_OTHERS');
Create a consolidate replay schedule
EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE
('visa_mcard_others_schedule');
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_VISA') FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_MCARD') FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_OTHERS') FROM dual;
EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;
Luís Marques - @drune - http://lcmarques.com
25. Consolidate Replay – Workload folding
Remap connections (DBA_WORKLOAD_CONNECTION_MAP) – You know how to
do it
Prepare your consolidation replay:
SQL> EXEC
DBMS_WORKLOAD_REPLAY.PREPARE_CONSOLIDATED_REPLAY
(synchronization=>’OBJECT_ID’, think_time_auto_correct => TRUE )
• Each capture subset have a different SCN. SCN-based sync is not possible
• Only OBJECT_ID and OFF are supported as syncronization options
• Connect_time_scale and think_time_scale were left at default
• Best Results avoiding connect time modifications or think time modifications
Luís Marques - @drune - http://lcmarques.com
26. Consolidate Replay – OBJECT_ID
A “more advanced” synchronization scheme is used
Ensures that SCN’s for different captures do not clash
Better dealing with concurrency when actions do not touch
the same object
Bugs happen:
If you find problems with OBJECT_ID sync you are left to no
syncronization
Monitor the progress of your replay. A collection of scripts
here help - Scripts to Debug Slow Replay (Doc ID 760402.1)
Luís Marques - @drune - http://lcmarques.com
27. Consolidate Replay – Workload folding
Start replay clients – At least one per subset
wrc system/oracle mode=replay
replaydir=/home/oracle/scripts/rat/db_replay_dir/
sleep 1
wrc system/oracle mode=replay replaydir=/home/oracle/scripts/
rat/db_replay_dir/
…
Start database replay
SQL> EXEC
DBMS_WORKLOAD_REPLAY.START_CONSOLIDATED_REPLAY;
Luís Marques - @drune - http://lcmarques.com
28. Review divergences after replay
Application Level divergences:
• Example: Check if all the VISA cards were processed (manual SQL Script)
Data Level divergences:
• Different number of rows returned by DML or SQL queries – Don’t match with capture
Error Level divergences:
• Errors on Capture that are not on replay
• Errors on Replay that were not captured
• Different errors on capture and replay
select DBMS_WORKLOAD_REPLAY.REPORT(replay_id => 1, format =>
'HTML') from dual;
Luís Marques - @drune - http://lcmarques.com
29. Sequential vs Parallel: Compare it!
Import AWR from previous sequential replay to compare to parallel replay
dbms_workload_replay.import_awr(replay_id => 12, staging_schema => 'PAY_A');
Generate a compare replays report
spool report_compare_consolidate.html
VAR v_clob CLOB
BEGIN dbms_workload_replay.compare_period_report (replay_id1 => 11,
replay_id2 => 12,
format => DBMS_WORKLOAD_REPLAY.TYPE_HTML, result => :v_clob );
END;
/
PRINT v_clob
spool off
Luís Marques - @drune - http://lcmarques.com
30. Sequential vs Parallel: Compare it!
Compare DB Time between periods (Sequential TP1 vs
Parallel TP2);
Easier than read an AWR
Luís Marques - @drune - http://lcmarques.com
31. Workload: Parallelized payment processing by Card Type
Goal: Parallelized payment processing by Card Type
and ATM machines payment processing from
another database capture
Scenario #2
Luís Marques - @drune - http://lcmarques.com
32. Consolidate Replay – Time shifting
Allows shifting a replay to a time in the future and align
with other replays
Stress testing by align peak loads
Peak number of sessions
Add another
workload here!
Luís Marques - @drune - http://lcmarques.com
33. Consolidate Replay – Time shifting
Align “Multibanco ATM processing” and Card Payment
Processing to start exactly at same time
VISA Payment
Process
MasterCard
Payment
Process
OthersCard
Payment
Process
Validations
Multibanco
ATM
processing
Workload
time
Workload
time
Delay time
Luís Marques - @drune - http://lcmarques.com
34. Consolidate Replay – Time shifting
Preprocess and delay the workloads to match ATM
replay
EXEC
DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE('payment_peak_align');
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM
dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_VISA', 600)
FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_MCARD', 650)
FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_OTHERS', 660)
FROM dual;
EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;
EXEC DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY
('payment_peak_replay', 'payment_peak_align');
Delay
secs
Luís Marques - @drune - http://lcmarques.com
35. Workload: Single workload capture on ATM machines payment
Goal: Stress test ATM machine payment processing on a
very busy day – December 24th
Scenario #3
Luís Marques - @drune - http://lcmarques.com
36. Consolidate Replay – Schema
Remapping
A single workload consolidated into two different instances of
the application
Original capture schema: MULTIBANCO_ATM (original)
Remapped schema: MULTIBANCO_ATM_B (remapped)
Luís Marques - @drune - http://lcmarques.com
Validations
Multibanco ATM
processing
Multibanco ATM
processing
Validations
Fees Calc
Fees Calc
MULTIBANCO_ATM_B
Workload
time
37. Consolidate Replay – Schema
Remapping
Add the same workload twice to a consolidate replay
SQL> EXEC
DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE(’multibanco
_stress_schedule');
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE
('DB_REPLAY_ATM') FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE
('DB_REPLAY_ATM') FROM dual;
EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;
EXEC
DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY
('payment_peak_replay', ‘multibanco_stress_schedule');
EXEC DBMS_WORKLOAD_REPLAY.SET_USER_MAPPING (2,
’MULTIBANCO_ATM', ’MULTIBANCO_ATM_B');
Luís Marques - @drune - http://lcmarques.com
38. Workload: Applied to all previous scenarios
Goal: Get a “even” more realist parallelized payment
processing
Scenario #4
Luís Marques - @drune - http://lcmarques.com
39. Query Only – Warm up my buffer cache
When you start your replay, remember that your
cache is "cold”
Replay may not be realistic or take too much time
– capture is done with already some blocks in buffer
cache.
Warm-up the buffer cache (if needed) with Query
Only Replay before the real workload.
Query Only Replay replays read-only statements
Luís Marques - @drune - http://lcmarques.com
40. Query Only – Warm up my buffer cache
example
EXEC
DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE
('visa_mcard_others_schedule');
SELECT
DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_
VISA’, query_only => TRUE) FROM dual;
SELECT
DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_
MCARD’, query_only => TRUE) FROM dual;
SELECT
DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_
OTHERS’,query_only => TRUE) FROM dual;
EXEC
DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;
Luís Marques - @drune - http://lcmarques.com
41. Final Notes
Consolidate Replay can be really helpful on stress
testing and scaling up on “brand new” PoC system
Comparing periods is important: Capture vs Replay;
Consolidate Replay vs Sequential Replays, etc
Follow capture and replay recommendations. Make
sure that you know your capture and his subsets and start
your replay with the “defaults”
Try different synchronization options if you find in
trouble with OBJECT_ID. Remember that OBJECT_ID
and OFF are the only options for Consolidated Replay
Luís Marques - @drune - http://lcmarques.com
42. Q & A
Luís Marques - @drune - http://lcmarques.com
Notes de l'éditeur
- Goals you set to archive. Discuss this with the management before start anything
Vendor didn’t approve the use of RAT for reproducing the workload
A step of day processing is using Oracle scheduler jobs, wich is a RAT limitation
Scope of the POC: Test a new platform from IBM P5 to Exadata; Stress test Exadata with real workload
Restrictions on Capture:
- Direct path load of data from external files using utilities such as SQL*Loader
- Flashback queries
- Distributed transactions - Any distributed transactions that are captured will be replayed as local transactions.
- Multi-threaded Server (MTS) and shared server sessions with synchronization mode set to OBJECT_ID
- Database scheduler Jobs
- A project to support Real Application Testing is on the way: We copied the production database, upgraded to 12c and start to do some tests that you will be see next.
direct path load,SQL*Loader, distributed transactions, database scheduler jobs, flashback queries, etc
5 Minutes of CPU Time – can be problematic on I/O bounded systems
Database will automatically switch to UNRESTRICTED after capture start
Only enables Oracle Database users with the RESTRICTED SESSION system privilege to connect to the database
DB_REPLAY_DIR must be empty and not in ASM – not supported
DB_REPLAY_DIR must be a shared directory on RAC
sts_capture_interval - Capture interval of the SQL set capture from the cursor cache in seconds
Capture files will contain all relevant information about the client request, such as SQL text, bind values, and transaction information.
USERNAME: Create a database user that can use Workload Intelligence ( a serious of grants are required)
JOB: Creates a new task to apply the algorithms of Workload Intelligence.
MODEL: Generates a model that describes the workload
PATTERNS: Read data from your capture and from the model you generated before (threshold probability –t that defines if a threshold will be part
REPORT: HTML pages that display the patterns found in previous stage (top 2– number of patterns to show up)
Offer you better context because SQL are not isolated.
You will know better your SQL and
Pre-processing must be done in PoC system – It can be an resource intensive job
Generate an HTML in your capture directory with the analysis name.
Consolidated Database Replay supports multiple workloads captured from one or multiple systems running Oracle Database 9i Release 2
A capture subset is a piece of a workload capture that is defined from an existing workload capture by applying a time range.
PARALLEL_LEVEL is documented but it is not supported as 12.1
OUTPUT_CAPTURE_DIR must be a subdirectory of INPUT_CAPTURE_DIR
Remap connections - This is mandatory as connection string will be pointing to Production
Replay clients replaydir must be your root directory
- Our Case: OBJECT_ID lots of problems because of concurrency, the objects touch the exact same table.
If you
- ATM processing does a few things before we have to wait until we can align everything up and start everything in parallel
Read Only Statements: SELECT, PL/SQL without DML and DDL or LOB reads, etc..