Case Study: Verizon - CA Workload Automation ESP Edition: Best Practice’s, DevOps, Change Management Utilizing Endevor

1. World®
’16
Case Study: Verizon - CA Workload Automation
ESP Edition: Best Practice’s, DevOps, Change
Management Utilizing CA Endevor®
Dave Kovaleski - MTS IV Specialist - Verizon
MFX138S
MAINFRAME AND WORKLOAD AUTOMATION

2. 2 @CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type
of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

3. 3 @CAWORLD #CAWORLD
Abstract
Verizon Telecommunications made CA Workload Automation ESP Edition (CA WA ESP) the
standard tool for workload management across the Wireline and Wireless organization for both
mainframe and distributed systems using the CA WA ESP agent technology. In this session, they
will share insights on how they successfully established CA WA ESP presence by instilling
Verizon's scheduling best practices, DevOps, and change management utilizing CA Endevor®.
Dave
Kovaleski
Verizon
MTS IV Specialist

4. 4 @CAWORLD #CAWORLD
Verizon Best Practices,
DEV-OPS, Change
Management Utilizing
CA Endevor®
Dave Kovaleski
November 16, 2016

5. 5 @CAWORLD #CAWORLD
Agenda
INTRODUCTION
SCHEDULING BEST PRACTICES
Q & A
DEV-OPS PROCESSING
CA WA ESP AND CA ENDEVOR®
SUMMARY AND CLOSING REMARKS
1
2
3
4
5
6

6. 6 @CAWORLD #CAWORLD
Introduction
Verizon IT uses best practices that support and enable a world class customer experience. The Production Control
department has implemented best practices which include using CA WA ESP global variables and symlibs as a central
repository to store application artifacts such as agent, path, and userid’s. The large applications processed by Verizon
benefit from the values that are used globally across applications. Other best practices include implementing naming
standards related to each application t hat support DEV-OPS solutions for application teams. Some of Verizon's application
teams perform their own application stop and starts. In addition, Verizon developers are now effectively using CA WA ESP
Web Services in a DEV-OPS tool. Verizon also has integrated the CA Endevor® Software Change Manager (CA Endevor®)
software to track application changes through the CA Endevor® software.

7. 7 @CAWORLD #CAWORLD
Experiences
Very positive feedback from
the application teams and
schedulers.
Comments Include: “Provide
one stop update for managing
server name changes” & “Easy
to migrate from test to prod by
updating variables in one
location!”
Best Practice
Defining and Creating Global
Variables or Symbolic libraries
to store agent, script name,
User ID information.
Findings
The use of Global Variables and
Symbolic Libraries provides a
user with a centralized location
to update and reference values
that will be utilized globally
throughout an application.
Best Practices
Global Variables and Symbolic Libraries

8. 8 @CAWORLD #CAWORLD
Best Practices
Mainframe Global Variable and Symlib(Member) Variable Name = ‘Value’
IPSNESP.CMD.CNTL.SYMLIB.VISN(CMDNS) AGENT='TWASTARDP'
FILE_PATH='/starload/DNS'
LOG_PATH='/starload/DNS/logs'
SCRIPT_PATH='/starload/DNS/process'
USERID='5staradm'
Global Variable and Symbolic Library Example

9. 9 @CAWORLD #CAWORLD
Best Practices
CA WA ESP Application and Job Definition Event Simulation
APPL CMD#DBCU
INVOKE 'IPSNESP.CMD.CNTL.SYMLIB.VISN(CMDNS)‘
UNIX_JOB DNSCM_BATCH_CBR_UPDATE
RUN DAILY
AGENT %AGENT
ARGS %SCRIPT_PATH./set_env_dns_callmanager.ksh >> +
%FILE_PATH./callmanager/batchCbr.log 2>&1
USER %USERID
SCRIPTNAME %SCRIPT_PATH./dnscm_batch_cbr_update.ksh
ENDJOB
INVOKE IPSNESP.CMD.PROCLIB.VISN(CMD#DBCU)
APPL CMD#DBCU
INVOKE 'IPSNESP.CMD.CNTL.SYMLIB.VISN(CMDNS)'
AGENT='TWASTARDP'
FILE_PATH='/starload/DNS'
LOG_PATH='/starload/DNS/logs'
SCRIPT_PATH='/starload/DNS/process'
USERID='5staradm'
UNIX_JOB DNSCM_BATCH_CBR_UPDATE
RUN DAILY
AGENT TWASTARDP
ARGS /starload/DNS/process/set_env_dns_callmanager.ksh >>
/starload/DNS/callmanager/batchCbr.log 2>&1
USER 5staradm
SCRIPTNAME /starload/DNS/process/dnscm_batch_cbr_update.ksh
ENDJOB
Global Variable and Symbolic Library Example

10. 10 @CAWORLD #CAWORLD
Experiences
Very positive feedback from
the application teams and
schedulers.
Comments Include: “I need to
focus on my workload and the
setup avoid distractions from
outside applications!”
Best Practice
Naming standards are defined
for applications, events and
procedure libraries.
Findings
The use of standardized names
allows our users to only view
and access their workload
when subscribing to workload
in the CA WA ESP Workstation.
The standards also allow ease
of monitoring and support for
both our operations and
application teams.
Best Practices
Application/Event/Proc Library Naming Standards

11. 11 @CAWORLD #CAWORLD
Best Practices
Event Naming Standard
Event names are defined as Event prefix.Event name. The event prefix name will be determined by what environment the application is running on.
The Event prefix is defined with the following format:
$VZWX AAA
Where:
$VZWX -The first 4 characters of the event prefix will always start with $VZWX
AAA - The last 3 characters (5-7) are defined as the 3 alpha Application Character Code
The Event name will be the application name that is triggered by the event.
Example:
$VZWXCMD.CMD#DBCU
| |
Event Prefix Application Name
Application/Event/Proclib Standardized Names Example

12. 12 @CAWORLD #CAWORLD
Best Practices
Application Naming Standard
Application names are limited to 8 characters and are defined with the following format:
AAAEXXXX
Where:
AAA - Characters are defined as the 3 alph character code
E - Characters are defined as the Environment Name
XXXX - Characters are defined as Free Form
Example:
CMD#DBCU
CMD # DBCU
| | |
Application Environment Free Form
Application/Event/Proclib Standardized Names Example

13. 13 @CAWORLD #CAWORLD
Best Practices
Mainframe Proclib Naming Standard
Mainframe Proclibs contain Application names are limited to 8 characters and are defined with the following format:
IPSNESP.AAA.PROCLIB.VISL
Where:
IPSNESP – Does not change
AAA - Characters are defined as the 3 alpha character code
PROCLIB – Does not change
VIS – Does not change
L – Mainframe Location Code
Example:
IPSNESP.CMD.PROCLIB.VISN
| |
Application North
Application/Event/Proclib Standardized Names Example

14. 14 @CAWORLD #CAWORLD
Experiences
Very positive feedback from
the application teams and
schedulers.
Comments Include:
“Controlling our workload
reduces downtime that our
customers may have incurred!”
Dev-Ops Option
An option was implemented by
the Scheduling team to allow
application teams to stop and
start all of their production
jobs. The scheduling technique
is heavily used when there is
scheduled maintenance.
Findings
By implementing an option to
stop and start their jobs, the
application team can control
and not rely on operations to
stop and start their workload.
This facilitates a DEV-OPS
solution to our application
teams
Dev-Ops
Application Team Dev-Ops Option Stop/Start Production Jobs

15. 15 @CAWORLD #CAWORLD
Dev-Ops
Step 1: Create Threshold Resource
RESDEF OTA_PRD_DATA_OTA_EASTBATCH ADD ENTERPRISE THRESHOLD MAX(1)
Stop and Start Setup

16. 16 @CAWORLD #CAWORLD
Dev-Ops
Step 2: Create Stop File Trigger Application
APPL OTAECFT
/* This appl contains all OTA EAST CFT jobs*/
FILE_TRIGGER OTAE_STOP_FT
/* Triggers OTA EAST Stop Event $VZWXOTA.OTAESTOP
RUN DAILY
AGENT OTAEAST
FILENAME '/ota/data/adm_ota/otaStopEastESP' CREATE NOCHANGE(1) CONTINUOUS($VZWXOTA.OTAESTOP)
DUEOUT EXEC 23:59 EASTERN
NOTIFY OVERDUE ALERT(GLB5)
Stop and Start Setup

17. 17 @CAWORLD #CAWORLD
Dev-Ops
Step 3: Create Stop Application
APPL OTAESTOP
/* This appl will set OTA_EASTBATCH to 0*/
JOB STOP.EASTOTA_BATCH TASK SELFCOMPLETING
/* This link job will stop the OTA East batch jobs.*/
RUN DAILY
RELEASE ADD(Cleanup_Stop_Start_Files.OTAEAST_STOP)
ESP RESDEF OTA_PRD_DATA_OTA_EASTBATCH SET AVAIL(0)
NOTIFY PNODE(COMPLETE) ALERT(SHTR)
ENDJOB
UNIX_JOB Cleanup_Stop_Start_Files.OTAEAST_STOP
/* This script will cleanup the ESP Stop/Start dummy files created to stop/start OTA batch
RUN DAILY
AGENT OTAEAST
USER adm_ota
SCRIPTNAME /ota/data/adm_ota/cleanupESP.sh
ENDJOB
Stop and Start Setup

18. 18 @CAWORLD #CAWORLD
Dev-Ops
Step 4: Create Start Continuous File Trigger Application
APPL OTAECFT
/* This appl contains all OTA EAST CFT jobs*/
FILE_TRIGGER OTAE_STRT_FT
/* Triggers OTA EAST Start Event $VZWXOTA.OTAESTRT
RUN DAILY
AGENT OTAEAST
FILENAME '/ota/data/adm_ota/otaStartEastESP' CREATE NOCHANGE(1) CONTINUOUS($VZWXOTA.OTAESTRT)
DUEOUT EXEC 23:59 EASTERN
NOTIFY OVERDUE ALERT(GLB5)
ENDJOB
Stop and Start Setup

19. 19 @CAWORLD #CAWORLD
Dev-Ops
Step 5: Create Start Application
APPL OTAESTRT
/* This appl will set the OTA_EASTBATCH to 1 when hold is released
JOB START.EASTOTA_BATCH TASK SELFCOMPLETING
/* This jobs will start all OTA East batch jobs.*/
RUN DAILY
RELEASE ADD(Cleanup_Stop_Start_Files.OTAEAST_START)
ESP RESDEF OTA_PRD_DATA_OTA_EASTBATCH SET AVAIL(1)
NOTIFY PNODE(COMPLETE) ALERT(SHTR)
ENDJOB
UNIX_JOB Cleanup_Stop_Start_Files.OTAEAST_START
/* This script will cleanup the ESP Stop/Start dummy files created to stop/start OTA batch
RUN DAILY
AGENT OTAEAST
USER adm_ota
SCRIPTNAME /ota/data/adm_ota/cleanupESP.sh
ENDJOB
Stop and Start Setup

20. 20 @CAWORLD #CAWORLD
Experiences
A very positive experience
throughout our application,
scheduling and auditing teams.
Comments Include: “Version
control is critical to business
resiliency as well as satisfying
audit requirements.”
CA Endevor®
With CA WA ESP
CA Endevor® has been
integrated with CA WA ESP to
track application updates and
manage job changes.
Findings
By implementing CA Endevor®,
the solution allows us to track
all application and job changes
for auditing purposes. The
technique also allows a quick
way to back-out changes made
to applications that may be
causing production issues.
CA Workload Automation ESP Edition and CA Endevor®
How CA Endevor® is Integrated With CA Workload Automation ESP Edition (CA WA ESP Edition)

21. 21 @CAWORLD #CAWORLD
CA WA ESP Edition and CA Endevor®
Step 1: Creation of the CA Endevor® Environment
The CA Endevor® Administrator in coordination with our Scheduling team defined the CA Endevor® environment. The CA Endevor® environment
contains the following: 1 Environment, 1 System, 27 Subsystems, 2 stages and 2 types (APL) for application and (SYM) for Global Variables/Symlib
members.
CA Endevor® Setup
Environment
EPSWLS
System
EPS
Subsystem
AAA00 (AAA = 3 Character
Application code)
Stage
Stage 1: PSTG
Stage 2: PROD
Type
Type 1: APL
Type 2: SYM
Step 2: Creation of the Package Shipment Rules
Since the CA Endevor® software is required to be installed on our development mainframe, we needed a process to send the updated applications to
the production mainframe proclibs. The CA Endevor® Administrator created Package Shipment rules to allow this to occur. Once the CA Endevor®
package is executed, the CA Endevor® with CA WA ESP elements are then ‘shipped’ to our production proclibs.

22. 22 @CAWORLD #CAWORLD
CA WA ESP Edition and CA Endevor®
Step 1: CA Endevor® ADD to Stage 1
CA Endevor® With CA WA ESP Edition Process

23. 23 @CAWORLD #CAWORLD
CA WA ESP Edition and CA Endevor®
Step 1: CA Endevor® ADD to Stage 1
CA Endevor® With CA WA ESP Edition Process

24. 24 @CAWORLD #CAWORLD
CA WA ESP Edition and CA Endevor®
Step 1: CA Endevor® ADD to Stage 1
CA Endevor® With CA WA ESP Edition Process

25. 25 @CAWORLD #CAWORLD
Step 1: CA Endevor® ADD to Stage 1
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

26. 26 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

27. 27 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

28. 28 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

29. 29 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

30. 30 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

31. 31 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

32. 32 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

33. 33 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

34. 34 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

35. 35 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

36. 36 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

37. 37 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

38. 38 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

39. 39 @CAWORLD #CAWORLD
Step 2: Package Creation and Shipment
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

40. 40 @CAWORLD #CAWORLD
CA WA ESP Edition and CA Endevor®
Step 2: Package Creation and Shipment Report Email
CA Endevor® With CA WA ESP Edition Process

41. 41 @CAWORLD #CAWORLD
Job Change Verification
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

42. 42 @CAWORLD #CAWORLD
Job Change Verification
CA WA ESP Edition and CA Endevor®
CA Endevor® With CA WA ESP Edition Process

43. 43 @CAWORLD #CAWORLD
CA WA ESP Edition and CA Endevor®
Job Change Verification and Event Simulation
CA Endevor® With CA WA ESP Edition Process

44. 44 @CAWORLD #CAWORLD
Summary
The Verizon Production Control department has implemented a myriad of options and scheduling
techniques to enhance our application users experience with the CA WA ESP Scheduling tool. The options
that were implemented are Best Practices with the use of Global Variables or Symlibs and Naming
Standards as well as DEV-OPS for end users to stop and start the production jobs during maintenance
windows and integrating CA Endevor® with the scheduler for version control management.
Verizon continues to roll out new Best Practices in the DEV-OPS arena plus Self Service options that will
permit application teams to hold or release batch jobs and the use of CA Endevor® for application teams
to track their lower environment changes within CA WA ESP.

45. 45 @CAWORLD #CAWORLD
Questions?

46. 46 @CAWORLD #CAWORLD
Thank you.
Stay connected at communities.ca.com

47. 47 @CAWORLD #CAWORLD
Mainframe and Workload Automation
For more information on Mainframe and Workload Automation,
please visit: http://cainc.to/9GQ2JI