7 Tasks You Should Automate in Oracle EBS
•Télécharger en tant que PPTX, PDF•
0 j'aime•41 vues
Download our presentation from Northwest Summit to learn the various areas of databases and applications where software bot automation can make a measurable difference. It addresses seven in particular that have been the most valuable in reducing the daily workload of our own DBAs and customers.
Recommandé
Recommandé
Contenu connexe
Plus de Datavail
Plus de Datavail (20)
Dernier
Dernier (20)
7 Tasks You Should Automate in Oracle EBS
- 1. 7 Tasks You Should Automate in Oracle EBS May 19, 2021 - 9:00 AM PT
- 2. www.datavail.com 2 Bachelor’s Degree in Computer Science More than 26 years’ experience in IT Technical background in • Implementing and supporting Oracle Applications with VLDB • Software Development • Oracle EBS • GoldenGate • Automation • Performance Tuning Vie Daranouvong Sr. Applications DBA
- 3. Fill Out Evaluation Form Fill out the session evaluation form for a chance to WIN an Oculus Virtual Reality Headset https://www.datavail.com/ ebs-webinar-survey/
- 4. www.datavail.com 4 About Datavail Our Range of Oracle Applications Services Strategic consulting 24x7 managed services Performance tuning Cloud migrations Assessments & roadmaps Monitoring Upgrades Data integration & ETL Applications & DB We Support: Oracle EBS Oracle Database Oracle Analytics GoldenGate WebLogic Oracle Data Integrator Cloud Platforms (AWS, Azure, Oracle Cloud) And more …
- 5. www.datavail.com 5 7 EBS Automation Tasks 7. Request Sets and Month-End 1. Clones and Refreshes 2. Daily Dashboard and Health Check 3. Concurrent Managers Monitoring and Alerting 4. Detecting Pending, Long Running, and Errored Concurrent Requests 5. Lingering Sessions 6. Performance Tuning
- 6. www.datavail.com 6 If you perform a task more than once, it should be automated. Automation is a key technology necessary for • Improving quality of service • Productivity • Efficient use of resource time Manufacturing and many other industries have been using robots for repetitive tasks for decades. As DBAs, we have software bots for similar purpose. We will discuss various areas of Oracle Applications software bots can help through automation. Why Automation?
- 8. www.datavail.com 8 1. Clones and Refreshes What are clones and refreshes? How long do clones and refreshes take to complete? Automation phases can include preparation, pre-clone, database tier clone, application tier clone, post-clone, and validation. Once a successful process has been established and tested many times, cloning and refreshes can be scheduled at the desired frequency.
- 9. www.datavail.com 9 1. Clones and Refreshes General Cloning Steps Create gold copy of DB and Apps files Pre Clone DB Tier and Apps Tier Target Cleanup DB Tier and Apps Tier Copy from source DB Tier and Apps Tier Configuration DB Tier and Apps Tier Postclone
- 10. www.datavail.com 10 Benefits 1. Clones and Refreshes Automation Details Manual Time (Hours) Automation Time (Hours) Time Save (Hours) Frequency (Monthly) Monthly Saved Hours Post Clone Truncate Tables 1.50 0.25 1.25 6.00 7.50 Clone Automation 14.00 4.00 10.00 2.00 20.00 EBS Clone Single Node 10.00 3.00 7.00 2.00 14.00 EBS Clone Multi Node 12.00 3.00 9.00 2.00 18.00
- 11. 2. Daily Dashboard and Health Check
- 12. www.datavail.com 12 18 Essential Proactive Checks 2. Daily Dashboard and Health Check 1. Database status 2. Tablespace usage 3. Alert log errors 4. Look for error in Apache/OACore/WebLogic Log files 5. Invalid objects & indexes 6. Database and archive log backup status 7. Concurrent manager status 8. Workflow mailer status 9. Application service status 10. Errored & Long-running concurrent requests 11. Objects with stale stats/gather scheme stats (GSS) status 12. OEM agent status 13. Cronjob alert executions 14. Create and maintain a runbook record changes and events 15. Alerts for long running user sessions (Toad, sql developer etc) 16. OEM performance page monitoring during business peak periods and load testing 17. Identify and report on worst-performing SQLs as candidates for tuning 18. Oracle SR monitoring and updates
- 13. www.datavail.com 13 You may have hundreds on systems to manage. To see system health, you may need to log into each system or OEM to see the health of each system and its components. Automating the check list by creating a report/dashboard for EBS that are sent in the beginning of the day will provide a snapshot of system health for 17+ components you your daily checklist can cut hours of investigation into minutes of review. These dashboards can be generalized with system summary for management and more comprehensive for system engineers and DBAs who would be more interested in the details. 2. Daily Dashboard and Health Check
- 14. www.datavail.com 14 2. Daily Dashboard and Health Check Hourly High-Level Dashboard HOST Status Summary: All HOSTS are up. Database Status Summary: All Databases are up.
- 15. www.datavail.com 15 2. Daily Dashboard and Health Check Customer Name EBS Summary
- 16. www.datavail.com 16 2. Daily Dashboard and Health Check RAC Instances RMAN Report Last 7 days Invalid Objects
- 17. www.datavail.com 17 Benefits 2. Daily Dashboard and Health Check Automation Details Manual Time (Hours) Automation Time (Hours) Time Save (Hours) Frequency (Monthly) Monthly Saved Hours Health check per environment 1.00 0.25 0.75 30.00 22.50 In addition to reducing human hours, this provides a proactive approach to catch issues ahead of time.
- 18. 3. Concurrent Managers & Workflow Monitoring and Alerting
- 19. www.datavail.com 19 A set number of concurrent managers defined and running ensures bandwidth to process requests in queue. Detecting and alerting on the number of actual running versus the maximum managers will help catch problem ahead of time. In addition to monitoring, alerting should be integrated with a ticketing system. 3.Concurrent Managers, Workflow Containers Monitoring and Alerting
- 20. www.datavail.com 20 3.Concurrent Managers& Workflow Monitoring and Alerting Concurrent Manger Details Workflow Component Services EBS Summary
- 21. www.datavail.com 21 Benefits 3. Concurrent Managers & Workflow Monitoring and Alerting Automation Details Manual Time (Hours) Automation Time (Hours) Time Save (Hours) Frequency (Monthly) Monthly Saved Hours Concurrent Manager Status 0.25 0.05 0.20 30.00 6.00 In addition to reducing human hours, this may avoid possible business impact.
- 23. www.datavail.com 23 When pending requests increase in queue or requests running longer than their average runtime, it may be an indication of potential problems. What is considered long running? Anything over 60 minutes? Anything running above their average runtime. Alert on pending and long running requests running over n% of average runtime. Notify business owners and/or users of issues with their requests. 4. Detecting Concurrent Requests: Pending, Long Running, Errored
- 24. www.datavail.com 24 4. Detecting Pending Requests Pending Requests
- 25. www.datavail.com 25 4. Detecting Long Running Requests Long Running Requests > 150% Avg Runtime
- 26. www.datavail.com 26 4. Detecting Requests Completed with Errors Concurrent Requests Error as of Midnight
- 27. www.datavail.com 27 4. Detecting Pending, Long Running, Errored Concurrent Requests Per problem incidents, analysis of the cause may take 30-45 minutes. Depending on the number of occurrences, time saving can vary. Through automation, issues are identified in real time which leads to faster resolution and customer satisfaction. Benefits
- 29. www.datavail.com 29 Reporting on these lingering inactive and orphaned session can help document and identify potential issues the sessions can cause. Known sessions that linger or become inactive may lock other sessions and consume system resources. There may be known users running certain forms/processes within certain time frame that do not terminate gracefully which may be candidates for automatic kill. The same is true for users/forms that may be session blockers. Automating killing of main and shadow lingering sessions will free up system resources and may alleviate performance issues. 5. Lingering Sessions
- 30. www.datavail.com 30 5. Lingering Session Per problem incidents, analysis of the cause may take 30-45 minutes. Depending on the number of occurrences, time saving can vary. Through automation, issues are identified in real time along with the culprit details which leads to faster resolution and minimizing impact window. Benefits
- 32. www.datavail.com 32 Monitoring and alert on SQLID that has changed its HASH value Monitoring and reporting on most optimal HASH for certain SQLID 6. Performance Tuning
- 33. www.datavail.com 33 Example SQL with baselines and “Fix” displayed in OEM via multiple steps versus automated scripts 6. Performance Tuning
- 34. www.datavail.com 34 Example SQL with baselines and “Fix” via automated scripts 6. Performance Tuning
- 35. www.datavail.com 35 SQL History sorted by best plan 6. Performance Tuning
- 36. www.datavail.com 36 6. Performance Tuning Per problem incidents, analysis of the cause may take 30-60 minutes. Depending on the number of occurrences, time saving can vary. Through automation, issues are identified at scheduled frequency along with the details which leads to faster resolution, minimizing impacted window, and restore system performance. Benefits
- 37. 7. Request Sets and Month-End Close Activities
- 38. www.datavail.com 38 7. Request Sets and Month-End Close Activities When components of a request set do not complete in their allotted time or failed, this affects other components downstream impacting month end timeline. Identifying and alerting components of requests sets that run longer that their allotted time or do not complete successfully will help catch potential problems ahead of time.
- 39. www.datavail.com 39 7. Request Sets and Month-End Close Activities Scheduled COMPLETE Data Collections Runtime MAIN Check and Status
- 40. www.datavail.com 40 7. Request Sets and Month-End Close Activities Scheduled COMPLETE Data Collections Runtime PARENT/Children Status Check
- 41. www.datavail.com 41 7. Request Sets and Month-End Close Activities Through automation, issues are identified at scheduled frequency which eliminates the need for user to actively monitor the request sets. This script becomes instrumental in identifying and traversing request hierarchy. Benefits
- 42. www.datavail.com 42 Honorable Mentions - Potential Automation Tasks Auto configure and standardization UNIX environment to create a consistent foundation for database and application installations. When deploying codes to development, stage, qa, and/or productions, ensure code consistency across multiple environments and spot differences at database object and file level. Code migration
- 43. Conclusion With automation and Dashboard, we can save between 15%+ of DBA time.
- 44. What should we do next? Set it and forget about it - Ron Popeil Next Steps
- 45. Fill Out Evaluation Form Fill out the session evaluation form for a chance to WIN an Oculus Virtual Reality Headset https://www.datavail.com/ ebs-webinar-survey/
- 47. www.datavail.com 47 Contact Us to Learn More 866-811-6764 info@datavail.com www.datavail.com Datavail Vie Daranouvong
Notes de l'éditeur
- A little but about myself. Started as developer/dba in 1994 at a consulting firm. Transitioned to doing full time APPS DBA 1996 and been supporting various aspects of EBS and Oracle database ever since. My specialties , just to name a new, are
- Before we start …. Please visit this URL and fill out a quick survey for a chance to win these awesome speakers. I’m told you can attach them to bamboo poles and use them as real tikis. Just leave them home when you go on your march.
- A little about datAvail, we provide these services for not just Oracle, but SQl Server, Postgres, mySQL, and others. Check out our website to know more.
- EBS, as you are aware, is a complex beast with many complex components. Managing these components is not an easy task. It takes time and often teams of people to feed it and support it. Anything to help ease these tasks will make life easier .. and isn’t that everyone’s goal. Making life somewhat easier for you, the DBA or support team AND for the customer. How we do this …. We can start with these 7 automation goals/tasks which HAS helped us at dataAvail to perform our tasks much more efficiently and provide additional value for our customers.
- Well that’s kind of obvious why … Repetitive/mundane tasks should be automated. Automation improves quality of service - it ensures consistent outcome and avoid human errors Automation helps with Productivity - it allows you to do more and be more productive It’s an efficient use of resource time - it frees you up for other tasks or more important less mundane tasks. Automation is not a new concept. It’s been around. Manufacturing have been using robots to produce goods and products. We have robots as well, software bots. What are bots? They are programs or sets of programs that perform their intended functions without intervention. These bots can be scheduled in crontab (Unix scheduling tool), via DBA_JOBS or DBMS_SCHEDULER in Oracle, through Application Scheduler as a concurrent request/sets, or manually initiated. Let us discuss where these bots can make a difference.
- A clone is an exact copy on an environment/system. With respect to EBS, it’s a copy of the database and software. These copies/clones are generally copies of production environments to create non-production environments like development, qa, stage , training, whatever.. Refreshes are renewing existing clones/environments. There are many steps to these clones and refreshes and these steps will have associated sub-steps. What I’m saying is that these activities are quite involved and take time. Clones and refreshes can take from 12 hours to multiple days to do manually. We had a case where refreshes took 12 hours of manual baby sitting, carrot feeding, eye on the screen making sure each step of the process is successful and fingers on the keys to initiate subsequent steps. It’s extremely time consuming AND exciting …. for some people Automation can cut 12 hours of cloning to 3. Even if time saving is not that significant, automating will free you from performing such mundane tasks. Since clones are so involved, you have to test and test until you have a fully working process. Once all the kinks are worked out, you can automate and schedule your daily, weekly, or monthly clones.
- This is just to outline the general steps involved. We can automate any or all parts of the clone on single node or multimode environment, various configuration and platform. The attached matrix shows us the BENEFITS … time saved for single environment. Imagine the time saved for doing 2-10 environments. Conversely, imagine the frustration and time wasted if we were to do these number of clones manually. As a managed service provider, we bill our customers for every hour worked. Time saved equals money saved for our customers.
- This is just to outline the general steps involved. We can automate any or all parts of the clone on single node or multimode environment, various configuration and platform. The attached matrix shows us the BENEFITS … time saved for single environment. Imagine the time saved for doing 2-10 environments. Conversely, imagine the frustration and time wasted if we were to do these number of clones manually. As a managed service provider, we bill our customers for every hour worked. Time saved equals money saved for our customers.
- As a DBA, one of your responsibilities is to know your system health and perform a daily check list to validate. What are these check list?
- It varies and dependent on what systems you are supporting. At a minimum, these should be performed on a daily basis. We’ve implemented these checks for all our customers… in addition to other checks we’re tasked with. It can be a starting point and include additional items as you see fit. with single environment .. it is easy and does not require too much time. when you have multiple .. it becomes time consuming and prone to error. How do make this easier and not miss crucial items?
- This is where a dashboard can help. Instead of manually logging into each server, then into each database to run your queries and performing the daily check list, a dashboard can be generated automatically with details ready for review .. within one report. From there it is only matter of identifying which item in the checklist/report needs attention and address them. The dashboard can be as simple or complex as you want. It can be a simple high-level overview of system statuses or a complex comprehensive report of all system components and processes. We’ll see some samples in the coming slides.
- This High-Level report provides a clear status for upper management who may only care to see this level of detail versus what DBAs would want to see more …
- This has a wealth of information providing details of the Database and Application in addition to what’s important in EBS. This report is a snapshot of system giving you baseline and metrics you can use for comparison. We’ve used information from such for help us with assessments and provide recommendations for our customers Notice the process and session count. It’ll be relevant in future slides.
- Additional information here provides issues and/or potential issues.
- The benefits here are many. Saves time, identify issues, hands free reporting, and prevent missed items for health check.
- In the EBS world, concurrent managers and workflow are the heart of the application. Concurrent managers are the key services for queuing and scheduling processes/requests. Workflow is a management system that automates and streamlines business process. If any component of these key services is down or not performing optimally, it could detrimentally impact business. Manually logging into EBS and navigating the menu system to get to the appropriate form and verifying these details takes time. Versus getting these details on the Dashboard and getting alerts when a service is down.
- What do we monitor and alert with respect to the Concurrent manager and Workflow. There are many concurrent managers defined. Each manager has a set number of worker(s) assigned. If a manager is assigned 10 workers, running with 9 workers signifies that a worker is terminated. Having less workers may cause requests to queue up on that manager as well slow down the number of requests that manager can run in parallel. No only would you want to alert, you wouldn’t want to be constantly checking emails for these alerts. Actionable alerts should be creating tickets in your ticketing system.
- This report shows a medium size environment with 30 managers in the Concurrent Manager Details. The Actual processes are highlighted in RED because the Assigned/Maximum process/worker does not equal to the Actual processes/worker running. In this case the Actual is greater that the Maximum. It’s not really an issue because this tells you that the manager is in the middle of a workshift change. A workshift is schedule allowing managers to grow or shrink through the day, When the ACTUAL is LESS than the MAXIMUM, then this is a real issue. Alerting should be trigger and root cause analysis should be performed. With respect to WorkFlow Components Services, any component down would impact business function and flow. This example shows the Wokflow Notification Mailer is down. It is a very important component if not most important. Imagine a PO workflow. When the mailer is down, no one is getting notified that’s there’s a PO that needs approval and cannot move alone in the flow. EBS Summary section provide general health and status.
- Like with previous slides, benefits are saving time and detecting issues.
- Why is this important? We can potentially catch problems before it becomes a bigger problem. Pending requests are requests waiting for a manager to let them in the queue to be processed. When you have more than the usual amount of waiting, it could signify a problem. With respect to long running, People have varying parameters when defining what long running requests are. Some requests are supposed to run longer than 60 minutes. We define long running as anything running over their average run time. In regards to requests with errors. Very likely, there are scheduled request that users and business owners may not actively monitor and see that their scheduled jobs have completed with errors. These may be business critical requests and the owners should be may aware so that they may take corrective actions. The next slides will show examples pending, long running , and errored requests.
- Some of these columns have been obfuscated for obvious reasons. The example shows just a handful of requests. When this list builds ups and hundred are in queue, it’s likely there’s an issues and requires investigation.
- This example of long running captures requests running over 150% of their average runtime. You may want to have additional logic to detect long running requests running over 150% of avg runtime 30 minutes. For certain user, responsibility, program, etc You have the tribal knowledge and understand your requests/programs and make that determination accordingly.
- Not that the previous slides were not as important, this is extremely important. As a DBA, if we catch and are aware of errors and issues before the users, then we are doing our job. It would be uneasy and embarrassing otherwise. I’m sure we’ve all been there.
- What are the benefits? Save time, faster resolution, keep customers happy.
- Oracle EBS does not always do a good job of cleaning itself up. When a user exits a form for various reasons, background sessions associated that form may not always terminate leaving it orphaned. In some cases, sessions may spawn many shadow sessions 5,10,20. This is especially true when running parallel threads.
- It’s important to detect, report, alert, and eventually kill these sessions. Consider it house cleaning. These sessions, although inactive, still consume resources (memory, connection pool, session count, process count). They may prevent other sessions from performing their tasks by holding a lock(s). There are cases where lingering inaction sessions would prevent new connections from connecting to the DB and APPS by maxing out SESSION and preventing existing session from processing by maxing out PROCESS. There were many instances where lingering sessions would bring the system to its knees and the database would lock up because log writers or archivers cannot process.
- You see a theme here… Automation : Saves time Detect Issues Minimize impact window
- This is complex subject and we’ll touch briefly on some things we can do to help with system performance.
- There may be a case where and acceptable execution plan/hash for certain SQLID have be identified, baseline created, but not “fixed”, detecting and alerting plan/hash change can report on newly created plan. The new plan may or may not be more efficient than the old. With this information, you can do a deep dive and determine the most optimal to fix in memory. We have seen with many customers that a HASH change can detrimentally impact performance to a point where the system becomes inoperable. In another cases where new code/SQLID is introduced, time is given for SQLID to bake and allow the optimizer time to come up with various HASHES for the SQLID Reporting on these plans/hashes can help identify the most optimal plan to use for the SQLID. The same report can be use not just with new SQLIDs, but old SQLIDs that have been overlooked or just need additional review/analysis.
- Manual check in OEM requires time to log in and navigate the tree menu , 1 Pull down Target to Database or Cluster 2 Choose the environment, 3 pull down Performance Hub, 4, scroll down through a list of hundreds of SQLID, click on the SQL 5. and find the right tab (Plan control) It’s time consuming to get to 1 SQL. AND … it doesn’t even tell you what HASH it is running
- Versus Automating this report and can report on ALL SQLIDs versus a single SQLIDs. Associated Hash Accepted Enable Fixed Number of execution You can add additional details ( average runtime)
- Versus Automating this report and can report on ALL SQLIDs versus a single SQLIDs. Associated Hash Accepted Enable Fixed Number of execution You can add additional details ( average runtime)
- Again .. Here Saves Time Restore System Performance Prevent Performance Impact
- Month-End Close Processes are extremely complex and requires involvement from various departments and owners of their respective modules. Tracking status of all the processes/request sets can take hours and coordination from various business owners. Month end close processes comprise many requests sets. These requests sets can be extensive and have children requests, grandchildren requests , and great grandchildren requests. Traversing these hierarchy for issues can be confusing and time consuming. Guest what? We can automate.
- Here is an example of just 1 request set in the entire scheme of month-end process. You see that the request set has errors at parent level, child level. 1 = parent = child = grandchild = great grandchilld
- Here you can see where all the errors are and notify business owners. We actually created this report that automatically goes to the business owners. In this example, they can see where the issues are at various point of the hierarchy tree would take appropriate actions defined in their playbook. This has saved the customer a lot of time and were extremely appreciative.
- Again Saves time Added great value customer Save user from manual investigation.
- I’ve mentioned 7 automation tasks, but there so much more you can do …. Auto Configure UNIX Layout foundation for db and application install, clone, refresh, update install all required RPM/packages -standardize oracle and applmgr environment Go further with ansible Code migration Simple form or pkg update To entire system release which could have days to migrate Save days to hours Comparing Code amongst various environment to ensure code consistency Code compare at SQL level And OS File level Security Audit Vulnerability Patch
- To recap, we’ve discussed some ideas you may not know you could automate and now you know. These automation ideas will improve quality of service avoid human errors efficiently manage the system helping with day-to-day operations catch potential problems ahead of time provide better customer support and at the end, make life somewhat easier. When done together, you can reduce total DBA time spent supporting the system by 15% or more
- Review these automation ideas … You may already have you own ideas brewing during this presentation … you’re welcome Assess the ideas and areas you can automate Prioritize bases on anticipated payback. You know, what gets you the most bang for your buck. Invest in efficiency – make automating a task a standard part of your routine Consider a 3rd party review There’s value in having a fresh set of eyes and ideas and often at little cost They can help improve security, performance, and help DBAs learn new skills, tips, tricks, and best practices Yes. We, at datAvail, can humbly offer assistance. Before we conclude, I’d like to leave you with a quote from a great philosopher: Set it and forget about.
- Before we start …. Please visit this URL and fill out a quick survey for a chance to win these awesome speakers. I’m told you can attach them to bamboo poles and use them as real tikis. Just leave them home when you go on your march.
- We’re opened to questions and discussion