Real life story about an APEX Application that suffered severe performance issues.

1. Copyright © 2013 Apex Evangelists
Troubleshooting APEX
Applications
Nienke Gijsen en Roel Hartman

2. About Roel
• Director of APEX Evangelists NL
• 20+ years of Oracle Experience
• “RoelH” in the APEX Forum
• Presenter at KSCOPE, OOW, Collaborate, DOAG, UKOUG, ...
• Oracle ACE Director
• Blog @ http://roelhartman.blogspot.com
• Twitter : @RoelH

3. Enterprise Manager

4. Alert.log
• Twee tot drie keer per week
• Apex process
• ORA-04030: out of process memory when trying to allocate 82456
bytes (pga heap,control file i/o buffer)
• ORA-04030: out of process memory when trying to allocate 16328
bytes (koh-kghu sessi,pmucalm coll)

5. Trace File
PRIVATE HEAP SUMMARY DUMP
15 GB total:
15 GB commented, 183 KB permanent
81 KB free (0 KB in empty extents),
15 GB, 1 heap: "session heap "
declare
rc__ number;
simple_list__ owa_util.vc_arr;
complex_list__ owa_util.vc_arr;
begin
owa.init_cgi_env(:n__,:nm__,:v__);
htp.HTBUF_LEN := -1;
null;
null;
simple_list__(1) := 'sys.%';
simple_list__(2) := 'dbms_%';
simple_list__(3) := 'utl_%';
simple_list__(4) := 'owa_%';
simple_list__(5) := 'owa.%';
simple_list__(6) := 'htp.%';
simple_list__(7) := 'htf.%';
simple_list__(8) := 'wpg_docload.%';
if ((wwv_flow_epg_include_modules.authorize('wwv_flow_file_mgr.get_file') = false) or (owa_match.match_pattern(p_string =>
'wwv_flow_file_mgr.get_file'

6. Metalink
PLSQL Procedure Causing ORA-04030: (pga heap,control
file i/o buffer) And ORA-04030: (koh-kghu sessi,pmuccst:
adt/record) or ORA-04030: (koh-kghucall ,pmucalm coll)
Errors [ID 1325100.1]
Solution
• Change the page count at the OS level:
• $ more /proc/sys/vm/max_map_count
$ sysctl -w vm.max_map_count=262144 (256K)

7. Apex Developers
• Error log
• Tijdstippen onderzoek
• Wat doet de get_file?
• Wat kan er 16GB groot zijn?

8. En
nu?
DBA

9. Troubleshooting APEX Applications
Debugging Tracing Logging

10. Debugging APEX Applications

11. Debugging APEX Applications
• Develop Mode
• Runtime Mode
• Websheets needs a Development environment !
• @apxdvins.sql / @apxdevrm.sql

12. Debugging Enable - Develop Mode
• Application Level Setting

13. Debugging Enable - Runtime Mode
begin
wwv_flow_api.set_security_group_id(p_security_group_id=>2616513429239104);! !
wwv_flow_api.set_enable_app_debugging(103,1);
end;
Workspace ID
Application ID
1 = ON
0 = OFF

14. Debugging - Develop Mode

15. Debugging - Runtime Mode
LEVEL1 ..
LEVEL9

16. Debugging - Instrumentation
apex_debug

17. Remote Debugging
• Using SQL Developer

18. Tracing APEX Applications

19. Tracing Enable - Develop Mode
• Workspace Level Setting (default “Yes”)

20. Tracing Enable - Runtime Mode
begin
apex_instance_admin.set_parameter( p_parameter => 'TRACING_ENABLED'
, p_value => 'Y' );
end;
SYS, SYSTEM, APEX_040200 or APEX_ADMINISTRATOR_ROLE

21. Tracing - Develop & Runtime Mode
DEBUGGING must be enabled as well!

22. Logging APEX Applications

23. Logging Enable - Develop Mode
• Workspace Level Setting (default “U”)
• Application Level Setting (default “Yes”)

24. Logging Enable - Runtime Mode
• Workspace Level Setting (default “U”)
begin
apex_instance_admin.set_parameter( p_parameter => 'APPLICATION_ACTIVITY_LOGGING'
, p_value => 'U' ); -- or [A]lways, [N]ever
end;
• Application Level Setting (default “Yes”)
begin
wwv_flow_api.set_security_group_id(p_security_group_id=>2616513429239104);!
wwv_flow_api.set_logging(103,‘YES’); -- Or ‘NO’ to switch it off
end;

25. Logging - Where Is It Going?
SELECT apex_view_name, comments
FROM apex_dictionary
where column_id = 0
AND apex_view_name LIKE '%LOG%'
•

26. Logging - How Long Is It In There?
• Workspace Level Setting
• Develop Mode
• Runtime Mode
begin
apex_instance_admin.set_log_switch_interval
( p_log_name in 'ACTIVITY' -- or ‘ACCESS’, ‘CLICKTHRU’, ‘DEBUG’
, p_log_switch_after_days => 180 ); -- between 1 and 180, default 14
end;
• Rotating Logs

27. Logging Examining - Develop Mode

28. Logging Examining - Runtime Mode
select application_id
, page_id
, to_char(view_date, 'DD-MM-YYYY HH24:MI') datetime
, log_context
, elapsed_time
, rows_queried
from apex_workspace_activity_log
order by elapsed_time desc

29. The Results
• Fluctuating Performance (avg 0.1 - max 1000 sec)
• Calls to WWV_FLOW_FILE_MGR.GET_FILE
• Used to retrieve images from the database

30. The Cause
•SELECT ‘<img src=”#WORKSPACE_IMAGES#no_picture.jpg”>’ ...
FROM <awesome join>
WHERE <more or less restrictions>
• Calls WWV_FLOW_FILE_MGR.GET_FILE tons of time
to retrieve ... nothing ....

31. The Solution
• Moved all images (and CSS / JavaScript) to web server
• Replaced ‘<img src=”#WORKSPACE_IMAGES#no_picture.jpg”>’
with ‘<img src=”/my_img/no_picture.jpg”>’
• Now using web server caching
• Database isn’t suffering from millions of calls anymore

32. Copyright © 2013 Apex Evangelists
http://apex-evangelists.com
Q& A

33. roel@apex-evangelists.com
http://apex-evangelists.com