"This presentations provides an overview of the most striking new functionality in Oracle Database 11g Release 2, as seen through the eyes of (database) developers. The presentation introduces new analytical functionality, the successor to the connect by clause for hierarchical queries, parallel statement execution, new packages and especially: Edition Based Redefinition. This presentation was performed (with live demos) during the AMIS Query on 29th September 2009.

1. Introducing Oracle Database 11g Release 2 for DevelopersOn SQL, PL/SQL and Application Development Lucas Jellema AMIS 29th September 2009

2. Quick Overview Analytical stuff LISTAGG, NTH_VALUE, Ignore nulls Recursive Subquery Factoring The new hierarchical query Parallel processing of statements Small Fry Big Fry: Edition Based Redefinition

3. Analytical Functions FIRST_VALUE and LAST_VALUE return the first and last value in a WINDOW For example a query to return the colleague (same department) who has the highest salarya similar query with last_value(or with order by salasc) would return the colleague with the lowest salary select deptno , ename , first_value(ename) over (partition by deptno order by saldesc) most_earning_colleague from emp NTH_VALUE complements FIRST_VALUE and LAST_VALUE

4. Analytical Function NTH_VALUE returns the NTH_VALUE in a WINDOW FIRST_VALUE(column) == NTH_VALUE(column,1) LAST_VALUE(column) == NTH_VALUE(column,1) FROM LAST For example a query to return the colleague (same department) who has the second highest salary select deptno , ename , nth_value(ename,2) from first over (partition by deptno order by saldesc) almost_highest_earning_colleague from emp NTH_VALUE complements FIRST_VALUE and LAST_VALUE

5. NTH_VALUE Measure_Expr and N are both expressions composed of columns, SQL functions, PL/SQL function call, constants, scalar subqueries FROM LAST can be used to start counting at the end of the window IGNORE NULLS can be used to ignore rows that have NULL as measure_expr

6. More on NTH_VALUE FROM FIRST (default) and FROM LAST can be used to start counting at either end of the window NTH_VALUE( sal ,2) over (order by saldesc) ==NTH_VALUE( sal ,2) FROM FIRST over (order by saldesc)==NTH_VALUE( sal ,2) FROM LAST over (order by salasc) ROWS UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING

7. More on NTH_VALUE IGNORE NULLS can be added to not count rows for which the expression evaluates to NULL The default is RESPECT NULLS NTH_VALUE(COMM, 2) IGNORE NULLSOVER (ORDER BY SAL DESC)means: Order the employees by salary, the top earner first Find the second row where COMM is not null Return the value of COMM in that row

8. Example: difference between top 2 For example a query to return all employees and per employee the difference between the highest earning colleague and the second highest earning colleague select deptno , ename , first_value(sal) over (partition by deptno order by saldesc) - nth_value(sal,2) from first over (partition by deptno order by saldesc) top2_difference from emp

9. LAG & LEAD – IGNORE NULLS In the same way as for NTH_VALUE, IGNORE NULLS is used to skip rows for which the measure is NULL This allows us to retrieve values from other rows in a more precise manner For example: Get the difference in COMM value between the current employee and the first less-experienced (later Hiredate) employee who also has a COMM value IGNORE NULLS (together with a CASE expression) gives us a row filter for the LAG, LEAD and NTH_VALUE functions

10. LAG without IGNORE NULLS Show the Salary of the next Hire select ename , sal , job , lead (sal) over (order by hiredate) sal_of_next_hire from emp order by hiredate

11. IGNORE NULLS & CASE– row filter Give me for each employee the salary of the next hire – but ignore CLERKs as they are in a league of their own select ename , sal , job , lead ( case when job ='CLERK' then null else sal end ) ignore nulls over (order by hiredate) sal_next_non_clerk_hire from emp order by hiredate

12. COLLECT aggregates records into collection create or replace type ename_type as table of varchar2(30) select deptno , avg(sal) avg_sal , cast ( collect(ename) as ename_type ) enames from emp group by deptno

13. LISTAGG – aggregating into string We already had COLLECT for aggregating multiple values into a Collection Requires a TYPE definition (usually TABLE OF VARCHAR2) Only Aggregate Function (group by), not Analytical Can do an order by especially for the Collected Values LISTAGG creates a delimiter separated string of values Produces a single VARCHAR2 (does not require a table type) Available as Aggregate (with GROUP BY) and Analytical Function (with PARTITION clause) Also includes an order by

14. select deptno , avg(sal) avg_sal , listagg( ename, ',') within group (order by sal) enames from emp group by deptno LISTAGG aggregates values into string

15. select deptno , ename , sal , listagg( ename, ',') within group (order by ename) over (partition by deptno) colleagues from emp order by deptno , ename LISTAGG aggregates values into string

16. Oracle goes industry standard…

17. Hierarchical Query Oracle specific (i.e. proprietary) approach Since Oracle 2 or 3 (very early on) Later extensions to hierarchical queries: order siblings by sys_connect_path connect_by_root, connect_by_is_leaf, nocycle, connect_by_is_cycle with employees (empno, name, level) as ( select empno, ename, job, mgr, level from emp start with mgr is null connect by prior empno = mgr ) select * from employees

18. Hierarchical Query in ANSI SQL ANSI SQL defines an alternative approach to Hierarchical Queries It is called Recursive Subquery Factoring (or Recursive With Clause) Apparently supported (to some extend) in DB2, SQL Server and PostgreSQL Extends WITH clause to enable formulation of recursive queries. “The query in the WITH clause (subquery) can refer to itself” Recursive WITH clause is ANSI. This makes Oracle ANSI-compatible for recursive queries.

19. Recursive Subquery Factoring with employees (empno, name, job, mgr, hierlevel) as ( select empno, ename, job, mgr, 1 from emp where mgr is null union all select e.empno, e.ename, e.job , e.mgr, m.hierlevel + 1 from emp e join employees m on (m.empno = e.mgr) ) select * from employees Root nodes Recursive Child nodes

20. What about the Connect by goodies? SYS_CONNECT_PATH Used to build the path from a node all the way to its root with employees (empno, name, mgr, hierlevel, path) as ( select empno, ename, mgr, 1, ename from emp where mgr is null union all select e.empno, e.ename , e.mgr, m.hierlevel + 1 , m.path||'/'||e.ename from emp e join employees m on (m.empno = e.mgr) ) select * from employees

21. What about the Connect by goodies? CONNECT_BY_ROOT Used to retrieve a value from a node’s ultimate root with employees (empno, name, mgr, hierlevel, root) as ( select empno, ename, mgr, 1, ename from emp where mgr is null union all select e.empno, e.ename , e.mgr, m.hierlevel + 1 , m.root from emp e join employees m on (m.empno = e.mgr) ) select * from employees

22. What about the Connect by goodies? ORDER SIBLINGS BY Set the sort order of child nodes under a parent node with employees (empno, name, mgr, hierlevel) as ( select empno, ename, mgr, 1 from emp where mgr is null union all select e.empno, e.ename , e.mgr, m.hierlevel + 1 from emp e join employees m on (m.empno = e.mgr) ) search depth first by name set seq select * from employees

23. Search Clause for organizing rows Use the SEARCH clause to specify ordering for rows Specify BREADTH FIRST BY if you want sibling rows returned before any child rows are returned. Specify DEPTH FIRST BY if you want child rows returned before any siblings rows are returned. Sibling rows are ordered by the columns listed after the BY keyword. The ordering_column is automatically added to the column list for the query name. Queries that select from the subquery can use an ORDER BY that refers to it

24. What about the Connect by goodies? ORDER SIBLINGS BY with employees (empno, name, mgr, hierlevel) as ( select empno, ename, mgr, 1 from emp where mgr is null union all select e.empno, e.ename , e.mgr, m.hierlevel + 1 from emp e join employees m on (m.empno = e.mgr) ) search breadth first by name set seq select * from employees order by seq

25. Recursive Subquery Factoring with employees (empno, name, mgr, hierlevel) as ( select empno, ename, mgr, 1 from emp where mgr is null union all select e.empno, e.ename, e.mgr, m.hierlevel + 1 from emp e join employees m on (m.empno = e.mgr) order by e.ename ) select * from employees

26. New PL/SQL Packages DBMS_DBFS_HS DBMS_COMPRESSION DBMS_DBFS_CONTENT DBMS_DBFS_CONTENT_SPI DBMS_DBFS_HS DBMS_DBFS_SFS DBMS_DST DBMS_PARALLEL_EXECUTE DBMS_STREAMS_HANDLER_ADM SEM_RDFCTX SEM_RDFSA

27. Some changes in PL/SQL packages dbms_session.GET_PACKAGE_MEMORY_UTILIZATION - memory usage instantiated packages dbms_space.ISDATAFILEDROPPABLE dbms_utility.GET_SQL_HASH dbms_utility.WAIT_ON_PENDING_DML Waits until all transactions (other than the caller's own) that have locks on the listed tables and began prior to the specified SCN have either committed or been rolled back

28. DBMS_PARALLEL_EXECUTE Run a statement in parallel on chunks of data Steps Create a task label Generate the chunks – rowid or value ranges Run the task: a SQL statement that has :start_id and :end_id bind parameters The task is executed with a certain degree of parallellism using job slaves

29. UTL_MATCH To find the similarity between strings Used for example in deduplication of records or finding fuzzy matches

30. Smaller Fry & Miscellaneous Create or Replace Type with FORCE option To replace even in the presence of dependents Automatic Function Result Cache reset (automatic data source detection) Function Result Cache accessible across RAC APPEND_VALUES Hint – to usewith INSERT INTO VALUES …. (especially PL/SQL FORALL) Fine Grained Invalidation extended to triggers

31. Smaller Fry & Miscellaneous Semantic Indexing of unstructured documents and the SEM_CONTAINS operator in SQL And many more new Semantic Technologies Handle Foreign Key dependencies for DBMS_FLASHBACK TRANSACTION_BACKOUT with CASCADE to roll back a transaction Internet Protocol version 6 (IPv6) Support IPv6 address has 128 bits, IPv4 address only 32 bits. IPv6 address must be in brackets in a URL

32. Smaller Fry & Miscellaneous IGNORE_ROW_ON_DUPKEY_INDEX and CHANGE_DUPKEY_ERROR_INDEX mandates To instruct Oracle to ignore duplicate keys (just skip the row) or report an ORA-38911 to find out just which UNIQUE KEY caused the exception Similar to 10gR2 DML Error Logging More setting up and restrictions for Unique Key on Update & Merge and Direct Path Insert

33. Release 2 Release 3 Base Release Introducing Oracle 11gR2 Editioningor:On Parallel Application Universes

34. Availability Availability = Performance * (Up or Down) Up = !Down Down = Unplanned Down + Planned Down Planned Down??? System Maintenance Power-supply, Hardware & Network, O/S upgrade Database patching & Upgrade Application Upgrades

35. Maximum Availability Architecture

36. Application Upgrade Creation of new objects Changing existing objects (alter and create or replace) Add, Modify or Drop columns or constraints Change packages and stored procedures Recompile Drop redundant objects Convert or migrate data Resume normal operations Application is DOWN

37. Compare with road maintenance

38. Restructuring A12 Build new road next to current one (A12-B) Same “functionality” as the old road At the cut-over moment Open new A12-B: Newly arriving traffic travels on temporary road Close A12 (original) Cars already on old road keep going

39. 11gR2 Editioning is similar Application Upgrade: Prepare new release Construct the release in parallel to the existing Operations in existing application ‘edition’ continue normally From the cut-over point: Have new sessions operate on new release Existing sessions can continue to run on existing release

40. 11gR2 Editions introduces a new dimension in the database Release 3 Release 2 Base Release

41. Editions introduce or inherit versions of objects Release 2 Release 3 Base Release

42. Editions are parallel universes Database Objects are identified by Object Type Schema Object Name …. Edition! (release, application version, stripe, parallel universe id) Database Sessions run in the context of a specific edition Using a specific collection of versions of objects

43. Database sessions run in a specific edition –one version of each object Release 2 Release 3 Base Release The database as seen by a sessionrunning in the context of Release 3 3 2 3 B 2 2 3 3 2

44. Demo of Application Upgrade Existing base application, in base edition Create new editon – release_2 Create and Alter database objects Base application continues running Cut-over point New sessions run in release_2 edition Current sessions continue in base

45. Some Rules for EBR (Edition Based Redefinition) Editioning acts on packages, functions, triggers, procedures, views, types and synonyms Editioning does not apply to tables Data is not versionend, cloned, migrated Different incarnations of a table are suggested through editionable views – there is only one table Applications should never access tables directly! Cross Edition Triggers on the table synchronize DML from different editions with the current state of the table CET are temporary objects – not part of the application

46. But wait, there is more After the release of a new edition – there is no reason why you cannot keep the previous one going for some time And multiple previous ones! That means – END OF THE BIG BANG upgrade! Multiple versions of the application can continue running to suport various user groups (e.g. SaaS) Without data migration and additional downtime upon later move over of user groups

47. Parallel Application Versions Application X VERSION 1 Application X VERSION 2 Release 2 Release 3 Base Release

48. Version 1 on Base Edition

49. Version 1 on Edition Release 2

50. Upgrade Base to Release 2 Authors New columns COUNTRY and BIOGRAPHY Books Drop column NUM_OF_PAGES Modified column ISBN (10 to 20 characters) New columns LANGUAGE and PUBLICATION_YEAR

51. Version 2 (on Edition Release 2)

52. Version 2 on Base Edition

53. Interesting Edition Tid-Bits Drop Object in an Edition stops the inheritance from previous Edition. Object no longer is carried forward Edition can have only one child – no branches (yet) DBMS_SQL.PARSE can be executed in a specific Edition Use parameter edition to specify other than current edition If no explicit edition is set for a session, the default edition is used ALTER DATABASE DEFAULT EDITION = edition_name; Data Dictionary Views DBA_/ALL_EDITIONS – editions in the database DBA_/ALL_OBJECTS – objects (inherited) in current edition DBA_/ALL_OBJECTS_AE – actual objects across all editions

54. Editions and Tables Tables cannot be versioned: there is only one definition of a table across all Editions Data is not versioned: a record exists once and only once Note: Workspace Management supports a form of versioning tables and data The solution for managing changes to Tables: Editioning Views and Cross Edition Triggers Editioning Views are defined on base table (no joins) Editioning Views can have DML triggers defined (just like base table) Using an editioning view in a query or DML statement does not impact the performance of the query or DML

55. Editions and Tables Application A Application B Table EMP SAL MGR JOB ENAME HIREDATE COMM

56. Editions and Tables Application A Application B Editioning View EMP Table EMP_BASE SAL MGR JOB ENAME HIREDATE COMM

57. Migrate to Editioning Views Rename Table (for example to <old table name>_BASE) (suggested but optional): changes the names of the columns in the table – to reflect change history For example JOB_1_0, JOB_1_1, JOB_1_2 … Create the Editioning View with the old table name Using the ‘CREATE OR REPLACE EDITIONING VIEW <view name>’ statement Recompile the triggers on the table These now become triggers on the Editioning View Recompile all invalid PL/SQL program units and Views They now refer to the Editioning View instead of the table VPD policies are reassigned to the View Regular auditing and FGA is on the table

58. Demo ALTER TABLE EMP RENAME TO EMP_BASE; CREATE OR REPLACE EDITIONING VIEW EMP AS SELECT ... FROM EMP_BASE / DROP TRIGGER EMP_BRI / Rem recreate trigger on Editioning View @<EMP_BRI>.trg Rem recompile invalid Views and PL/SQL units

59. Multiple versions of Editioning View Application A Application B Edition R2 Edition R1 Editioning View EMP Editioning View EMP Table EMP_BASE SAL MGR JOB ENAME HIREDATE COMM

60. Multiple versions of Editioning View Application A Application B Edition R2 Edition R1 View EMP (1.1) …* Language View EMP (1.0) Table EMP_BASE SAL MGR JOB ENAME LANGUAGE HIREDATE COMM Forward CrosseditionTrigger

61. Demo CREATE EDITION R2 AS CHILD OF R1; ALTER SESSION SET EDITION R2; SELECT SYS_CONTEXT('Userenv', 'Current_Edition_Name') "Current_Edition" FROM DUAL; ALTER TABLE EMP_BASE ADD (LANGUAGE VARCHAR2(2) NULL); Rem function for deriving value for language CREATE OR REPLACE FUNCTION GET_LANGUAGE( p_job in varchar2) return varchar2 is begin return case p_job when 'MANAGER' then 'fr' else 'en' end; end;

62. Demo Rem Create Forward Crossedition Trigger Rem Applies to DML operations on EMP_BASE from Rem Earlier Editions CREATE OR REPLACE TRIGGER EMP_1_1_Fwd_Xed BEFORE INSERT OR UPDATE ON EMP_BASE FOR EACH ROW FORWARD CROSSEDITION DISABLE BEGIN :new.language = get_language(:new.job); END EMP_1_1_Fwd_Xed; Rem Enable the Forward Crossedition Trigger ALTER TRIGGEREMP_1_1_Fwd_Xed ENABLE;

63. Demo Rem Use Forward Crossedition trigger to Rem upgrade existing table records according to new table Rem version (for large # records use dbms_parallel_execute) DECLARE c NUMBER := DBMS_SQL.OPEN_CURSOR(); x NUMBER; BEGIN DBMS_SQL.PARSE ( c => c , Language_Flag => DBMS_SQL.NATIVE , Statement => 'UPDATE EMP_BASE SET EMPNO = EMPNO' , Apply_Crossedition_Trigger => 'EMP_1_1_Fwd_Xed' ); x := DBMS_SQL.EXECUTE(c); DBMS_SQL.CLOSE_CURSOR(c); COMMIT; END;

64. Upgrade Table Definition (Create Edition,) Switch to Edition Make desired changes to the table Add columns, modify Columns, … Modify the Editioning View in the Edition To reflect the table as its latest version should look Perhaps hiding columns you eventually want to drop (optional) Modify Editioning Views in previous Edition(s) If the table change invalidates those Views (optional) Create Forward Crossedition Trigger on base table to have DML on previous Editions made valid (optional) Create Reverse Crossedition Trigger on base table to have DML on current Edition made valid

65. Cross Edition Triggers If you remove a (mandatory) column from the current Editioning View… a Reverse Crossedition Trigger ensures that new records get some value in that (invisible) column If you add a (mandatory) column to the table (and the current Editioning View)… a Forward Crossedition Trigger ensures that records DMLed through previous Editioning View versions are made valid (optionally) Apply Forward Crossedition Trigger for all existing records (created in old edition of the table) Use DBMS_SQL.parse (with parameter Apply_Crossedition_Trigger set to name of trigger) Use DBMS_PARALLEL_EXECUTE

66. Multiple versions of Editioning View Application A Application B Edition R3 View EMP (1.1) … (minus ENAME)* Language o FIRST_NAME * LAST_NAME Edition R1 Edition R2 View EMP (1.1) …* Language View EMP (1.0) Table EMP_BASE Reverse Crossedition Trigger SAL MGR JOB ENAME LANGUAGE FIRSTNAME LASTNAME HIREDATE COMM Forward CrosseditionTrigger

67. “Suggested (best) Practices” Every application (release) sets the Edition it requires when it connects to a session At the same time it calls dbms_application_info And sets other Context details Applications should never access tables – all access should be through views Only through views can the data structure itself be Editioned Even triggers should be on the Editioning View

68. Summary 11gR2 Editions are parallel, co-existing universes with incarnations of database objects The new release can be constructed, tested and run in a new edition The old edition can be switched off at cut-over Editions also allow long time co-existence of multiple releases of an application Application Upgrade no longer needs to disrupt the operation through planned downtime

69. Conclusion See http://www.amis.nl To do an 11gR2 PoC together with us To learn about 11gR2 training (DBA & Developer) For more on database Migration services See Blog for 11gR2 articles http://technology.amis.nl/blog Contact me: lucas.jellema@amis.nl