Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Jenkins Online Meetup - Automated SLI based Build Validation with Keptn

This presentation was given as part of the Jenkins Online Meetup

  • Soyez le premier à commenter

Jenkins Online Meetup - Automated SLI based Build Validation with Keptn

  1. 1. Automating SLI/SLO based build validation with Keptn and Jenkins Andreas Grabner DevOps Activist at Dynatrace @grabnerandi https://www.linkedin.com/in/grabnerandi https://www.meetup.com/Jenkins-online-meetup Part of #jenkinsOnlineMeetup Star us @ https://github.com/keptn/keptn Follow us @keptnProject Tutorials @ https://tutorials.keptn.sh Thu, June 4th @ 3PM GMT+1 Hands-On @ https://github.com/keptn-sandbox/jenkins-tutorial
  2. 2. 2Confidential Use Case: Automate Build Approvals Through SLI/SLO-based Quality Gates
  3. 3. Confidential 3 Challenge #1:Lengthy manualapproval Build Deploy to „Test“ Run Test In „Test“ Manual Approval Promote to „Staging“ Deploy to „Test“ Functional: Test Result Trend Not Enough Performance: Manual Comparison Is Slow Monitoring: Too much unstructed data Which build is better Which data comes from my testIs this a real problem ~30-60min
  4. 4. 4Confidential Inspirations to solve this problem
  5. 5. Confidential 5 Inspiration #1 –Google‘s SREPractices • Service Level Indicators (SLIs) • Definition: Measurable Metrics as the base for evaluation • Example: ErrorRate ofLogin Requests • Service Level Objectives (SLOs) • Definition: Binding targets forService Level Indicators • Example: Login ErrorRate must be less than 2% • Service Level Agreements (SLAs) • Definition: Business Agreement between consumer andprovidertypically based on SLO • Example: Logins must be reliable & fast (ErrorRate, Response Time, Throughput) 99% within a 30 day window • Google Cloud YouTubeVideo • SLIs, SLOs, SLAs, oh my! (class SRE implements DevOps): https://www.youtube.com/watch?v=tEylFyxbDLE SLIs drive SLOs which inform SLAs
  6. 6. Confidential 6 Inspiration #2 –ThomasSteinmaurer‘s implementationinContinuousDelivery “Performance Signature” for Build Nov 16 “Performance Signature” for Build Nov 17 “Performance Signature” (SLA) aggregated for every Build “Multiple Metrics” (SLIs) e.g: Memory, Errors … for specific timeframe Simple Evaluation (SLOs) Against a threshold or prev build
  7. 7. Confidential 7 SLI/SLO-basedevaluationimplementationinKeptn SLIs definedperSLI Provider as YAML SLIProviderspecificqueries,e.g:DynatraceMetricsQuery QualityGates ... Dynatrace Prometheus Neoload Scores SLIs Queries SLI Providers with SLI Definitions & Timeframe SLOs definedon Keptn ServiceLevelas YAML Listofobjectiveswithfixedorrelativepass& warncriteria indicators: error_rate: "builtin:service.errors.total.count:merge(0):avg" count_dbcalls: "calc:service.toptestdbcalls:merge(0):sum" jvm_memory: "builtin:tech.jvm.memory.pool.committed:merge(0):sum" objectives: - sli: error_rate pass: - criteria: - "<=1“ # We expect a max error rate of 1% - sli: jvm_memory - sli: count_dbcalls pass: - criteria: - "=+2%" # We allow a 2% increase in DB Calls to previous runs warning: - criteria: - "<=10" # We expect no more than 10 DB Calls per TX total_score: pass: "90%" warning: "75%" 0.5 1.0 0.0 info 7/8 (87.5%) 4/8 (50%) $ keptn start-evaluation 30m myservice sli.yaml slo.yaml 5 DB Calls 360MB 4.3% 123SLI Value: SLI Score: Total Score 2 3 4 Tool X 1
  8. 8. Confidential 8 Exampleexplainer onSLI/SLO-basedEvaluation Overall Failure Rate Source: Dynatrace Query: builtin:service.errors.total Response Time LOGIN Tx Source: Dynatrace Query: calc:service.responsetime:filter(tx, LOGIN) # DB Calls LOGIN Tx Source: Dynatrace Query: calc:service.dbcalls:filter(tx, LOGIN) <= 5% <= 2% <=150ms & <=+10% <= 400ms <= +0% <= +1 Build 1 0% 80ms 100ms SLO: Overall Score Goal 90% 75% Response Time 95th Perc Source: Dynatrace Query: builtin:service.responsetime(p95) <=100ms <= 250ms SLOSLIs (Service Level Indicators) warn pass 3 100% Build 2 4% 120ms 90ms 3 75% Build 3 1% 90ms 120ms 6 62.5% Build 4 0% 95ms 95ms 3 100% Build 1 Build 2 Build 3 Build 4 $ keptn send event start-evaluation myproject myservice starttime=build1_deploy endtime=build1_testsdone$ keptn send event start-evaluation myproject myservice starttime=build2_deploy endtime=build2_testsdone$ keptn send event start-evaluation myproject myservice starttime=build3_deploy endtime=build3_testsdone$ keptn send event start-evaluation myproject myservice starttime=build4_teststart endtime=build4_testsend
  9. 9. Confidential 9 EvaluationResult intheKeptns Bridge SLO Result per SLI OneBuild/Eval Total Score per Build/Evaluation Values per SLI Values across builds/evaluations Heatmap Chart
  10. 10. Confidential 10 Solution #1:AutomateApprovalthroughSLI/SLO-basedQuality Gates Build Deploy to „Test“ Run Test In „Test“ Manual Approval Promote to „Staging“ Deploy to „Test“ Trigger Quality Gate Wait for Result SLI & SLO Result: success, Score: 85/100 Run Test In „Test“ w Tagging Rt(p95) < 500ms #ofSQLs <= 5 cpu(max)< 80% Java GC < 2% ... T a g g i n g Pull SLIs for Testing time frame Validate SLOs Build Deploy to „Test“ Promote to „Staging“ Deploy to „Test“~1min ~30-60min
  11. 11. 11Confidential Setting up Keptn to be used byJenkins Mydemo will use Dynatrace Using Prometheus? https://tutorials.keptn.sh/?cat=prometheus
  12. 12. Confidential 12 Step #1:InstallKeptnwithMonitoring Support for Dynatrace:tutorials.keptn.sh api.keptn.* bridge.keptn.* Evalulation Testing Deploy Notify ... ... ... ... $ keptn install (full or quality gate) $ keptn configure monitoring dynatrace Pre-Requisits k8s or Ubuntu Linux (for microk8s) Dynatrace Tenant, API & PaaS Token Installation on k8s/OpenShift Installation on Ubuntu https://tutorials.keptn.sh/?cat=microk8s GitOps Dev Staging Production Project: perfproject (repo) • Stage: performance (branch) /Service: perfservice (folder) slo.yaml dynatrace/sli.yaml jmeter/mytest.jmx API API Push Events (Start Test, Evaluation Done ...) Pull Metrics (SLI) Configuration (Tagging Rules, Dashboards ...) $ keptn add-resource sli.yaml, slo.yaml, ... $ keptn create project perfproject Setup a Keptn Project (will automate through Jenkins) $ keptn create service perfservice
  13. 13. Confidential 13 Step #2:Instrument yourappenv andprepare yourtests docker run -p 80:8080 yourregistry/yourcontainer I n f r a P r o c e s s S e r v i c e T e s t homepage version echo invoke homepage version echo invoke Monitor the enviornment your CI/CD Deploys to deploy Kubectl apply –f yourapp.yaml java –jar yourapp.jar CI/CD Envs
  14. 14. Confidential 14 Step #3:Create SLIs & SLOs basedon e.g:yourdashboards Take the metrics & tags that work for you And list them in your SLI.yaml including filter for Tag and Test Step And start with a simple SLO.yaml, e.g: within 10% of last build
  15. 15. Confidential 15 Nowwe areready to haveKeptn automatetheanalysis forusandintegrate it inJenkins Instead of manually comparing timeframes 1 2 3 4 1 2 3 4 x 1 2 3 4 x Keptn automates that process based on SLIs & SLOs X
  16. 16. Confidential 16 Demo#1:based onhttps://github.com/keptn-sandbox/jenkins-tutorial 1 2 3
  17. 17. Confidential 17 Demo#1:SLI/SLO Evaluationas aSelf-Service aka„Validate yourSLIs are working“ Keptn Init Trigger Quality Gate Wait for Result init "qgproject" "qualitystage" "evalservice" "dynatrace" addResource "dynatrace/sli.yaml" addResource "slo.yaml" addResource "dynatrace/dynatrace.conf.yaml" sendStartEvaluationEvent "600" waitForEvaluationDoneEvent GitOps Project: qgproject (repo) • Stage: qualitystage (branch) /Service: evalservice (folder) slo.yaml dynatrace/sli.yaml dynatrace/dynatrace.conf.yaml builtin:service.requestCount.total:merge(0):sum &entitySelector=tag($SERVICE),type(SERVICE) builtin:service.requestCount.total:merge(0):sum &entitySelector=tag(evalservice),type(SERVICE) &from=now-600s&to=now Each SLI will be transformed & pulled Result: failure, Score: 60/100 dynatrace.com/api/v2/metrics/query/transformedSLI Keptn Jenkins Library $ keptn add-resource sli.yaml, slo.yaml, ...$ keptn create project qgproject (qualitystage)$ keptn create service evalservice $ keptn configure monitoring dynatrace qgproject Quality Gate Result Event via dynatrace.com/api/v2/events
  18. 18. Confidential 18 Demo#2:based onhttps://github.com/keptn-sandbox/jenkins-tutorial 1 2 3
  19. 19. Confidential 19 Demo#2:AutomateSLI/SLO Evaluationfor existing Tests executed inyourJenkinsPipeline Keptn Init Trigger Quality Gate Wait for Result init "testwithqgproject" "qualitystage" "testservice" "dynatrace" addResource "dynatrace/sli.yaml" addResource "slo.yaml" addResource "dynatrace/dynatrace.conf.yaml" sendStartEvaluationEvent waitForEvaluationDoneEvent GitOps Project: testwithqgproject (repo) • Stage: qualitystage (branch) /Service: testservice (folder) slo.yaml dynatrace/sli.yaml dynatrace/dynatrace.conf.yaml builtin:service.requestCount.total:merge(0):sum &entitySelector=tag($SERVICE),type(SERVICE) builtin:service.requestCount.total:merge(0):sum &entitySelector=tag(testservice),type(SERVICE) &from=startoftest&to=now Each SLI will be transformed & pulled Result: failure, Score: 60/100 dynatrace.com/api/v2/metrics/query/transformedSLI Keptn Jenkins Library Run Simple Load Test markEvaluationStartTime -> Trigger your tests Quality Gate Result Event via dynatrace.com/api/v2/events
  20. 20. 20Confidential Use Case: Performance-Driven Culture Provide Performance as a Self-Service
  21. 21. Confidential 21 Challenge #2:AutomatingPerformance with Jenkinsrequires managingtools &environments Build Deploy to „Test“ Standup Test Infrastructure Gather Performance Data How to enable different workloads? How much hardware is needed to run these tests?Where do we run these tests? Execute Tests Evaluate Results Where do we stream test metrics to? How to analyze the results? Who manages this infrastructure? DIY (Do It Yourself) approach: lots of online guides available!
  22. 22. Confidential 22 Challenge #2:AutomatingPerformance with Jenkinsrequires managingtools &environments Build Deploy to „Test“ Standup Test Infrastructure Gather Performance Data Execute Tests Evaluate Results Build Deploy to „Test“ + Notify Keptn Wait for Result SLI & SLO Result: success, Score: 85/100 Rt(p95) < 500ms #ofSQLs <= 5 cpu(max)< 80% Java GC < 2% ... Pull SLIs for Testing time frame Validate SLOs Test Scripts + Workload
  23. 23. Confidential 23 Demo#3:based onhttps://github.com/keptn-sandbox/jenkins-tutorial 1 2 3
  24. 24. Confidential 24 Demo#3:Performance asa Self-Service Keptn Init Trigger Quality Gate Wait for Result init "perfaasproject" "performance" "perfaasservice" "dynatrace" addResource "dynatrace/sli.yaml" addResource "slo.yaml" addResource "dynatrace/dynatrace.conf.yaml" sendDeploymentFinishedEvent "http://appundertest.local" waitForEvaluationDoneEvent GitOps Project: testwithqgproject (repo) • Stage: qualitystage (branch) /Service: testservice (folder) slo.yaml dynatrace/sli.yaml dynatrace/dynatrace.conf.yaml requestCount:tag($SERVICE) ,type(SERVICE) requestCount: tag(perfaasservice), type(SERVICE) &from=startoftest&to=now SLI Magic (short) Result: failure, Score: 60/100 Keptn Jenkins Library addResource "jmeter/load.jmx" addResource "jmeter/jmeter.conf.yaml" jmeter/load.jmx jmeter/jmeter.conf.yaml Testing url: http://appundertest.local Quality Gate Result Event via dynatrace.com/api/v2/events Deploy & Test Events via dynatrace.com/api/v2/events
  25. 25. Confidential 25 Additionallyinstalled Keptn Services inmydemos • JMeterExtended Version • Supporting multiple workload configurations via jmeter.conf.yaml • https://github.com/keptn-contrib/jmeter-extended-service
  26. 26. 26Confidential Keptn Jenkins Shared Library https://github.com/keptn-sandbox/keptn-jenkins-library
  27. 27. Confidential 27 Using theLibrary: fullreadme on https://github.com/keptn-sandbox/keptn-jenkins-library • 1:Add it in yourJenkinsGlobal Settings • 2:ConfigureKeptn Properties: KEPTN_API_TOKEN,KEPTN_ENDPOINT,KEPTN_BRIDGE • 3:Add it to yourpipeline
  28. 28. Confidential 28 Current Plansfor the Keptn JenkinsLibrary • Support Credentials vs EnvironmentVariables for Keptn Tokenand Endpoint • Potentially createa JenkinsPlugin to visualizeHeatmap in Jenkins • Provide callbacks to Keptn whenKeptn calls JenkinsPipelines as part of end-2-enddelivery orchestration • Extension ofhttps://github.com/keptn-sandbox/jenkins-service
  29. 29. Automating SLI/SLO based build validation with Keptn and Jenkins Andreas Grabner DevOps Activist at Dynatrace @grabnerandi https://www.linkedin.com/in/grabnerandi https://www.meetup.com/Jenkins-online-meetup Part of #jenkinsOnlineMeetup Star us @ https://github.com/keptn/keptn Follow us @keptnProject Tutorials @ https://tutorials.keptn.sh Thu, June 4th @ 3PM GMT+1 Hands-On @ https://github.com/keptn-sandbox/jenkins-tutorial
  30. 30. Confidential 30 Q&A &ActionItems • Please STARour projects on GitHub • https://github.com/keptn/keptn • https://github.com/keptn-sandbox/jenkins-service • https://github.com/keptn-sandbox/keptn-jenkins-library • https://github.com/keptn-sandbox/jmeter-extended-service • JoinourSlack Channel • https://github.com/keptn/community • Tryour tutorials: • https://tutorials.keptn.sh • https://github.com/keptn-sandbox/jenkins-tutorial • Giveus feedback and feel freeto contribute 
  31. 31. 31Confidential Bonus Use Case: Break the Monolithic Pipeline Let Keptn orchestrate your „Jenkins Micro-Pipelines“ using https://github.com/keptn-sandbox/jenkins-service
  32. 32. Confidential 32 Most monoliths are NOT enabled for „Cloud Native“ Cloud Native
  33. 33. Confidential 33 Mixedinformationabout • Process(build, deploy,test,evaluate,…) • Targetplatform (k8s, …) • Environments(dev,hardening,…) • Tools(Terraform,Helm,hey,…) No clear separationof concerns • Developers • Define which artifact to use • Want fast feedback on their code • DevOpsEngineers • Define which tools to use • Ensure tools areproperly configured • SiteReliabilityEngineers • Define delivery processes • Define operations workflows 33 It starts with pipelines whoare built like the monolithic counterparts!
  34. 34. Confidential 34 The experience weare looking for? Not this! But this!
  35. 35. Confidential 35 Removehard dependencies and integrations Build Prepare Deploy Test Notify Rollback Config Mgmt. Deploy Test Monitoring ChatOps Rollback
  36. 36. Confidential 36 Removehard dependencies and integrations Build Prepare Deploy Test Notify Rollback Config Mgmt. Deploy Test Monitoring ChatOps Rollback Eventing Event:Deploy Artifact:container1 Stage:Dev Strategy:Blue/Green
  37. 37. Confidential 37 Removehard dependencies and integrations Build Prepare Deploy Test Notify Rollback Eventing Event:Test URL:myservice Stage:Dev Strategy:Performance which events to generate who consumes events Config Mgmt. Deploy Test Monitoring ChatOps Rollback
  38. 38. 38Confidential How Keptnaddresses this problem
  39. 39. Confidential 39 Eventing Keptn is built on an architecture that separates concerns Application Plane Define overall process for delivery and operations (stages, deployment, tests, remediation) Control Plane Follow application logic and communicate/configure required services Deploy Service Test Service Validation Service Monitoring Service Config Service Remediate Service Artifact / Microservice API & UI Site Reliability Engineer DevOps Developer shipyard.yaml uniform.yaml
  40. 40. Confidential 40 Use Jenkins-Service https://github.com/keptn-sandbox/jenkins-service 1. Follow the readme on GitHub 2. Wait for next Keptn Webinar  for a deep dive jenkins/jenkins.conf.yaml

×