Contenu connexe


When Management Asks You: “Do You Accept Agile as Your Lord and Savior?"

  1. Dan Lagos “AdmFord" - Cyphercon 6 - 2023 When Management Asks You: “Do You Accept Agile as Your Lord and Savior?” Agile in Operations, can it actually be done?
  2. What is Agile in the first place? The Principles • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale • Business people and developers must work together daily throughout the project • Build projects around motivated individuals. Give them the environment and support they need, and trust them to ge the job done • The most e ffi cient and e ff ective method of conveying information to and within a development team is face-to-face conversation • Working software is the primary measure of progress • Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace inde fi nitely • Continuous attention to technical excellence and good design enhances agility • Simplicity — the art of maximizing the amount of work not done — is essential • The best architectures, requirements, and designs emerge from self-organizing teams • At regular intervals, the team re fl ects on how to become more e ff ective, then tunes and adjust its behavior accordingly
  3. Origins of Agile From the Factory Floor, to the O ffi ce Floor • Theory of Constrains Any system is limited from achieving more of its goals by a small number of constraints, and by focusing to identify and restructure the organization around them, we limit their impact. 1. Identify the constraints (bottlenecks) 2. Decide how to exploit the constraints 3. Subordinate everything to the above decision 4. Elevate the system’s constraints 5. If the constraint breaks (is eliminated), go back to step 1. But don’t allow inertia to cause a new constraint • TOC advocates keeping inventory of products low. • You don’t make new inventory when you can’t sell it • You keep in-work inventory low by using small batches • Time to Process an element of inventory is key to fi nding where bottlenecks appear • Inventory is anything that requires labor to create, or modify
  4. Origins of Agile The Toyota Way • Continuous Improvement • Challenge Long term vision for company/program • Kaizen Continuous Improvement • Gench Genbutsu Go to the source to fi nd facts to make correct decisions • Respect for people • Respect Respect others, and take responsibility to do our best to build mutual trust in a team • Teamwork Stimulate personal and professional growth, share opportunities of development and maximize individual and team performance • Right Process • Create a process that can bring problems to the surface • Use a pull system to avoid overproduction • Level out the workload (don’t overload workers) • Build a culture where it’s ok to stop a process in order to fi x it, when problems are identi fi ed • Standardize tasks and processes • Use visual control so no problems are hidden • Use already tested technology • Add value to the organization • Grow leaders who understand the work, live the philosophy, teach it to others • Develop people and teams who follow the company’s philosophy • Respect partners and suppliers by challenging and helping them to improve • Continuously solve root problems drives organizational learning • Go see for yourself the situation to understand it • Make decisions by consensus, considering all options, then implementing the solution quickly • Become a learning organization through re fl ection and continuous learning
  5. It’s basically project management But without the project managers (to a point) • Every Plan Fails when Confronted with the Enemy • The further you plan out work, the more of a chance unexpected work or circumstances will wreck your schedule • Waterfall works when plans don’t change • Increasing Flexibility Means Breaking work up into smaller more manageable chunks • Iteration based on the chunk size, work with team leads, management, and employees to identify issues and solutions quickly.
  6. Methodology Letter Jumble SCRUM, KANBAN, SCRUMBAN, etc… SCRUM KANBAN SCRUMBAN Team Members Recommended members between 3-9 No speci fi c limitation on the number of team members No Speci fi c Limitations on the number of team members Team Roles Members are assigned di ff erent roles & responsibilities Members are generalists or specialists No Roles Work Cycles Sprints that can last from 1 to 4 weeks Continuous Work fl ow 2-Week Iterations with continuity (the board is not cleared) Rules Follows Strict Rules Relaxed and Flexible Finds the middle grounds between scrum & kanban with moderate rules Task Assignments Assigned to the team members Members choose their fast- paced tasks Members choose their tasks Limits Based on the current sprint Limits placed on the work-in- progress Limits placed on the work-in- progress
  7. Scrum Meetings The Structured Work Process (That can, or usually be, a PITA) Frequency Attendees Time Needed Objectives Sprint Lasts anywhere from 1 to 4 weeks Sprint Planning First day of sprint Product Owner, Scrum Master, Scrum Team Potentially an hour, depending on team size Set a sprint goal Daily Scrum Daily Scrum Master & Scrum Team No more than 15 minutes What did you do yesterday? What are you doing today? Any Blockers/Issues? Sprint Review At the end of a sprint Scrum Team, Product Owner, Scrum Master 2 - 4 hours Demo of work done user stories - con fi rm and decide on incomplete ones Assesement of Backlog Sprint Retrospective Between Review & Planning meetings Scrum Team, Scrum Master, Product Owner An hour What was done well What didn’t go as planned Improvements for next sprint Backlog Re fi nement Every other week depending on the duration of the sprint Scrum Master, Product Owner, Potentially Scrum Team No de fi ned duration Prioritize backlog items Alight backlog to KPIs Appropriate sizing of backlog items Add more detail
  8. Agile Is Not Simply Implementing a Methodology Stay away from “Cargo Cult Agile” • An Organization that wants to implement “Agile" can’t just order the use of Scrum or Kansan across the board. • Every Operations Team will have di ff erent organizational, and business requirements • it’s a two way street • Methodologies can be implemented partially, or in a hybrid manner (SCRUMBAN) • But institutional organization also must change to give Project Managers/ Product Owners / Team Leads more initiative to initiate & lead changes
  9. Methodologies and Team Evolution Using Scrum to Grow your Team • Methodologies, while not necessary to fully implement, can be used to help in team evolution. • Stages of Team Development • Forming & Storming • More structured work environment is bene fi cial • Norming • As people get used to working together & with their tools, the structure can be eased as needed • Performing • Less outright structure is needed as team members know what they are supposed to do and are active in choosing task to complete • Adjourning, or changes in structure • Other than closing teams, people will also come and go. So it can be useful to re-implement more structure from methodologies to get identify issues and corrective measures.
  10. Actually Putting Agile to work, DevOps The Culmination of Agile in the Development World • In development, the culmination of Agile can be said to be DevOps. • DevOps has the same principles as Agile, while not necessarily following strict methodologies. • It de fi nes objectives regarding the types of work and techniques to achieve them • It expects good knowledge of what the team members are doing, and creating/implement tools that they can implement changes frequently. • We see this in things like the Software Development Lifecycle (SDLC)
  11. Types of Work and Priorities The Four Types of work • Gene Kim in the Phoenix Project and DevOps handbook de fi ned the four types of work in a business • Business Projects (Highest Priority) • Business Initiatives, most of development work • Internal IT Projects (Highest Priority) • Infrastructure and IT Operations • Updates and Changes (Normal Priority) • Often Generated from the two previous types of work • Unplanned Work or Recovery Work (should be limited as much as possible) • Incidents and problems generated by other work
  12. The Three Ways of DevOps • Systems Thinking/Flow of Work • All work should ideally fl ow in one direction, such as across a KANBAN board • Any fl ow back in the process means that there are unidenti fi ed potential issues or constraints that need to be addressed • Amplify Feedback Loops • Learn from current processes and fi nd improvements • Implement improvements to the processes • Experimentation • Try multiple ways of improving processes and work and test which work better • In development, the best example of this is A/B testing
  13. DevOps foundations in Security (DevSecOps?) Enterprise vs MSSP • Enterprise & Team Projects take priority • Maintenance and Operations work (tasks for other teams, installation of software, etc) is lower priority • Incidents take engineers away from improvements and should be limited. • 80% Project & Ops work 20% Incident work • Automate Incidents and Operations to meet KPIs and team SLAs • MSSP Improvements, Maintenance & Operations work may be handled by a separate team compared to Security Monitoring • Incidents are the main work done • 80% Incident work 20% identifying improvements • Based on Agreement with other Organizations, apply automations and process improvements to meet KPIs and SLAs
  14. Starting the Change to Agile People, Process, Tools • Change has to come from both the top (Senior Management & Management) and within the teams themselves • Identify all the processes that your team is involved with. That is tasks received from other teams. Incidents your team have de fi ned and how to resolve them. Common operations work that is done frequently • SOAR is your friend. It is never too early to start. Playbooks in Word, Sharepoint, OneNote, or Wiki are fi ne. But a playbook within your ticketing system (such as the SecOps module in ServiceNow), forces analysts to follow the work fl ow, ensuring repeatability. This also creates the foundations for automating the Incident Response process.
  15. Constraints and Bottlenecks Time is literally money • IT operations often doesn’t have many bottlenecks regarding infrastructure, that is when certain commands/tools taking too much time. But there is one bottleneck that is universal. • Man/Woman-hours of work. • Don’t expect 8 hours of fully productive work from a person • Lose 2 hours for lunch, breaks, and meetings • Consider the fi rst hour and last hour as “half productivity”, as they’re getting up to speed for the day, or thinking what needs to get done when they get back home • 4 hours of fully productive work, 2 hours at 50% productivity, 2 hours of zero productivity • Find the people that are always being called to put out fi res, or take on the most work for themselves.
  16. From Process to Pipeline Identify the work process, fi nd improvements, rinse and repeat 1. Knowing your team’s work processes is foundational. Since it is literally understanding what work is done, and how it is done. 2. Interactive Playbooks are a fi rst step to standardize work 3. Identify dependencies (where other teams are needed) and constraints (certain tasks always falling to a single team member) 4. Implement improvements A. Automate parts of the playbooks B. Request that other teams automate some of their work that’s related to your own team’s 5. If Incident Response to alerts has been fully automated, create new alerts based on frequency of the automated ones, to fi nd discrepancies to normal daily fl ow 6. New services, products and technologies are implemented regularly, so go back to step 1
  17. Software Development Life Cycle (SDLC) The foundation behind development and DevOps 1. Planning 2. Analysis 3. Design 4. Implementation 5. Testing & Integration 6. Maintenance
  18. Software Development Lifecycle Adapting the Dev process into SecOps 1. Planning: Identify IOC that you want to create an alert for 2. Analysis: Research IOC 3. Design: Identify availability of log sources to see if IOC would be visible 4. Implementation: Write the initial query/alert for the SIEM, ideally in a code repository with version tracking 5. Testing & Integration: Test the query in the SIEM for existing indicators from log archive A. If successful, push query to SIEM as a new alert rule B. If unsuccessful, push query to SIEM with limited alert generation (potential new alerts are only assigned to the user) C. Assign Purple Team member to write test article on dedicated machine to con fi rm alert will actually fi re. 6. Maintenance: Run regular “drills”, making sure that all know alerts are able to detonated at will by the team to make sure everything works correctly (think, " fi re drills”) SIEM
  19. Ticket Metrics The Subtle Art of Of Proving a team’s Work • Number of tickets closed is a disingenuous metric. It takes in consideration the di ffi culty or the result (true positive or false positive). • Automatically assigning tickets to users randomly is good, but you can’t use the count of how many tickets are closed to measure employee performance. • Pick and pull of tickets at discretion also doesn’t work, as employees with experience can identify potentially easy tickets and horde them. • Classifying tickets based on priority level (Critical, High, Medium, Low) and assigning a points value to them can help, but frequency of occurrence of the tickets and the results (true or false positives) still aren’t considered. Allowing employees to potentially game the results in their favor. • But SCRUM o ff ers something else, “Planning Poker”, assigning a weight for a task based on the amount or di ffi culty of work to complete it. Assigning a separate weight to True Positive incidents, and False Positives can normalize the amount of work done between team members.
  20. Planning Poker Stealing From Scrum, to Make Metrics Work • Planning Poker is usually done during backlog re fi nement. • Values are decided on how much work a task takes, and NEVER how much time it would. People in general cannot give good estimates of the amount of time things will take. But we’re more visually oriented so measuring things by estimating size or quantity can actually be more accurate. • Planning Poker uses a modi fi ed fi bonacci sequence to estimate work. 0, 1, 2, 3, 5, 8, 13, 20, 40, 100 • In doing so, you do not take in consideration SLAs or KPIs • Track these separately, as adjacent goals to achieve • The law of averages in the end comes to our rescue regarding metrics
  21. x̄ and σ Average and Standard Deviation in a Normal Distribution • In a Normal Distribution, the average (mean), x̄, is at the central point of the curve • The standard deviation, σ, measures how dispersed the data is compared to the average (mean)
  22. Out of Bounds Tickets When Response times exceed Standard Deviation (both Positive & Negative) • When they take more time • Dependencies can be to blame, team members are waiting for other teams to complete associated work • Employee needs additional training or is a new team member and needs to improve • There potentially is a problem with a tool the team uses • These can be the fi rst targets to quickly improve SLA results by automation • When they take less time • An employee has found a way to automate their work • Congratulate them, and see about standardizing their procedure! • An Employee has been closing tickets by only doing the work partially. • Retraining might be required • Closer monitoring of employee’s work • Termination of employee
  23. Track the Averages Track Team Progress • The average time it takes for a job with a certain Estimation of E ff ort (Planning Poker value), even when not linked to an KPI or SLA, can provide valuable information on the team. • As the team evolves (forming, storming, norming, performing), the average time it takes for work should drop month to month. • This includes performance improvements done with automations and improvements done in conjunction with other teams. • Time saved = Money Saved. Doing more tickets in less time and improved e ffi ciency means management can o ff er a measure of money saved over time to senior leaders at an organization. • Changes to teams can raise these averages, seeing how fast they start dropping again can indicate how well the team is working together.
  24. Useful Books Some Study Materials • Foundational Texts • "The Goal: A Process of Ongoing Improvement” by Dr. Eliahu Goldratt • “Toyota Kata” by Mike Rother • “The DevOps Handbook” by Gene Kim • Useful in my opinion, as Security can also be a creative enterprise, I would also suggest the following book: • "Creativity, Inc.: Overcoming Unseen Forces That Stand in the Way of True Inspiration" by Ed Catmull
  25. Certs? When you need some credentials • Scrum? • Useful to have if you’re doing Scrum Master work. • Professional Scrum Master (PSM) by • No need to take a class. Higher minimal passing score needed. pushes more for understanding how to use scrum and adapt it to your requirements. • Certi fi ed Scrum Master (CSM) by Scrum Alliance • Requires you to take a class. Comparatively low passing score. The exam concentrates more on wrote knowledge of Scrum, and not as much how to apply or adapt it. • Agile • Anything potentially from PMI • If you don’t have the work history for a PMP • Try the CompTIA Project+ • Or do the Certi fi ed Associate in Project Management (CAPM) by PMI
  26. Thank You Twitter: @admford Mastodon: Linkedin: