4K - Kubernetes with Knative, Kafka and Kamel
•
0 likes•3,196 views
Staring 2019, we started to see “Serverless” taking center stage in all application architectures with enterprise integration and data streaming being no exception. With Kubernetes becoming the standard to run your application workloads in cloud, there isn’t’ one simple way to build highly scalable platform to integrate streaming data and enterprise applications without compromising enterprise compliance. In this session we will explore the four pillars Kubernetes,Knative,Kafka and Kamel to know how they help you build the much needed platform for enterprise integration and data streaming which can scale to infinity without comprising on your cloud resource costs. Presented as part of Cloud Community Days on 19th June - ccdays.konfhub.com
Recommended
Recommended
More Related Content
More from CodeOps Technologies LLP
More from CodeOps Technologies LLP (20)
Recently uploaded
Recently uploaded (20)
4K - Kubernetes with Knative, Kafka and Kamel
- 1. bit.ly/kubemaster1 1 4K - Kubernetes dn.dev/4k-kubernetes with Knative, Kafka and Kamel
- 3. dn.dev/4K-kubernetes Kamesh Sampath(devadvocate.yaml) apiVersion: bio/v1 kind: DeveloperAdvocate spec: title: “Director of Developer Experience @ Red Hat” gde: expert: cloud year: 2019 javaDeveloper: true opensourceContributions: - Eclipse Che - Quarkus - Supersonic Subatomic Java - kubernetes/minikube: Run Kubernetes locally - Camel K social-handles: linkedin: https://linkedin.com/in/kameshsampath github: kameshsampath (Kamesh Sampath) twitter: Kamesh Sampath (@kamesh_sampath) blogs: developers.redhat.com(Kamesh Sampath) kubectl email --address kameshs@redhat.com
- 4. dn.dev/4k-kubernetes Agenda Understand four pillars of Cloud Data Streaming ○ Kubernetes ○ Knative ○ Apache Kafka ○ Camel-K(Kamel)
- 6. dn.dev/4k-kubernetes Love Thy Mono Old School New School
- 7. dn.dev/4k-kubernetes A Challenge Server Hardware Operating System Application Server .war or .ear Java Virtual Machine Custom Configuration Linux Kernel Version & Distribution Java 1.6.6_45 or Java 1.7.0_67 Weblogic 10.x.y, Tomcat 6.x.y, JBoss EAP 6.x.y JDBC driver, datasource, JMS queue, users Have you ever had “/” vs “” break your app? Or perhaps needed a unique version of a JDBC driver? Or had a datasource with a slightly misspelled JNDI name? Or received a patch for the JVM or app server that broke your code? Containerize Your App
- 8. dn.dev/4k-kubernetes Email MyApp.war has been tested with the following On my Windows 7 desktop JDK 1.8.43 Wildfly 9 Configuration: Datasource: MySQLDS Tested with: mysql-connector-java-5.1.31-bin.jar Production Environment Red Hat Enterprise Linux 6.2 JRE 1.7.3 WebSphere 8.5.5 Oracle 9
- 9. dn.dev/4k-kubernetes Dockerfile Container Guest OS Java App Server custom configuration FROM centos/wildfly COPY xyz.xml /opt/myserver/configuration/ COPY mysql-connector-java-5.1.31-bin.jar /opt/wildfly/standalone/deployments/ COPY mysql-sample-ds.xml /opt/wildfly/standalone/deployments/ COPY myapp/target/your.war /opt/wildfly/standalone/deployments/ your.war dependencies
- 10. dn.dev/4k-kubernetes DevOps Challenges for Multiple Containers ▪ How to scale? ▪ How to avoid port conflicts? ▪ How to manage them on multiple hosts? ▪ What happens if a host has trouble? ▪ How to keep them running? ▪ How to update them? ▪ Rebuild Container Images? Node Node Node Node Node Logger Node
- 12. dn.dev/4k-kubernetes ▪ Greek for “Helmsman,” also the root of the word “Governor” (from latin: gubernator) ▪ Container orchestrator ▪ Supports multiple cloud and bare-metal environments ▪ Inspired by Google’s experience with containers ▪ Open source, written in Go ▪ Manage applications, not machines Meet Kubernetes
- 13. dn.dev/4k-kubernetes History of Microservices Continuous Integration via XP 1999 AWS EC2 2006 DropWizard May 2011 Agile Manifesto Feb 2001 NETFLIX to AWS 2010 Ribbon March 2012 Hystrix March 2012 Eureka July 2012 Microservices Assess Thoughtworks Radar March 2012 Spring Boot Sept 2013 Microservices Defined Thoughtworks Fowler, Lewis March 2014 Kubernetes June 2014 Java EE6 2009 DevOps 2009 Docker March 2013 Vert.x June 2011 2 0 1 5
- 16. dn.dev/4k-kubernetes Pods A group of whales is commonly referred to as a pod and a pod usually consists a group of whales that have bonded together either because of biological reasons or through friendships developed between two or more whales. In many cases a typical whale pod consists of anywhere from 2 to 30 whales or more.* *http://www.whalefacts.org/what-is-a-group-of-whales-called/
- 18. dn.dev/4k-kubernetes Pod Replicaset/ Deployment Service Label ✓ 1+ containers ✓ Shared IP ✓ Shared storage (ephemeral) ✓ Shared resources ✓ Shared lifecycle ✓ The Desired State - replicas, pod template: health checks, resources, image ✓ Grouping of pods (acting as one) has stable virtual IP and DNS name ✓ Key/Value pairs associated with Kubernetes objects (env=production) Kubernetes Terms Persistent Volume ✓ Network available storage ✓ PVs and PVCs
- 19. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Nodes kubelet kubelet kubelet kubeletkubeletkubelet
- 20. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Declarative kubelet kubelet kubelet kubeletkubeletkubelet image: repo/mytomcat:v1 replicas: 4
- 21. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - 4 Tomcats kubelet kubelet kubelet kubeletkubeletkubelet
- 22. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Pod Fail kubelet kubelet kubelet kubeletkubeletkubelet X
- 23. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Correcting kubelet kubelet kubelet kubeletkubeletkubelet
- 24. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - Node Fail kubelet kubelet kubelet kubeletkubeletkubelet X
- 25. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Dev Ops api etcd scheduler controllers kubelet kubelet kubeletkubeletkubelet Kubernetes Cluster - Pods Replaced
- 26. dn.dev/4k-kubernetes Master Node Node Istio proxy Node Node Node Dev Ops api etcd scheduler controllers Kubernetes Cluster - New Node kubelet kubelet kubeletkubeletkubelet Node kubelet
- 27. dn.dev/4k-kubernetes Serverless https://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-3.pdf
- 29. dn.dev/4k-kubernetes Serverless Computing “Serverless computing refers to the concept of building and running applications that do not require server management. It describes a finer-grained deployment model where applications, bundled as one or more functions, are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment.” The CNCF takes steps toward serverless computing
- 30. dn.dev/4k-kubernetes Why we need Serverless ?? Agility in any cloud environment Event driven cloud native applications Focus on business differentiation Consistent and scalable operations Resource and Cost Optimization
- 31. dn.dev/4k-kubernetes Serverless Computing “Serverless computing refers to the concept of building and running applications that do not require server management. It describes a finer-grained deployment model where applications, bundled as one or more functions, are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment.” The CNCF takes steps toward serverless computing
- 32. dn.dev/4k-kubernetes Short History of Serverless AWS S3 March 2006 AWS Lambda November 2014 AWS EC2 August 2006 Serverless Described Thoughtworks August 2016 Serverless Coined October 2012 Serverless Assess Thoughtworks Radar April 2016 AWS Dynamo DB January 2012 Firebase September 2011 Iron.io July 2011 Google Cloud Function Feb 2016 Azure Functions Mar 2016 Pivotal Riff Dec 2017 Oracle Functions Sept 2016 IBM Bluemix OpenWhisk Feb 2016 Oracle Fn Project Oct 2017
- 33. dn.dev/4k-kubernetes Architectural Styles Services Microservices Serverless • Autonomous • Loosely Coupled • YourControl • Known Programming Model • Very Matured • Single Purpose • Stateless • Independently Scalable • Automated • Your/Cloud Control • Known Programming Model • Matured • Ephemeral • Cloud Control • Short-Lived Processes • New Programming Model • Event-Driven Async • Immature
- 34. dn.dev/4k-kubernetes When to choose what ? Services Microservices Serverless • Autonomous • Loosely Coupled • Single Purpose • Stateless • Independently Scalable • Automated • Single Action • Ephemeral Your Control Cloud Control Portability
- 35. dn.dev/4k-kubernetes Microservices Serverless Functions Your Control Long-Lived Processes Known Programming Model Often Sync Request-Response Mature: IDE Integration Debuggers Tracers Monitoring CI/CD Cloud Control Short-Lived Processes New Programming Model Event-Driven Async Immature: ?
- 36. dn.dev/4k-kubernetes Microservices Serverless Functions Your Control Long-Lived Processes Known Programming Model Often Sync Request-Response Mature: IDE Integration Debuggers Tracers Monitoring CI/CD Cloud Control Short-Lived Processes New Programming Model Event-Driven Async Immature: ? Knative
- 38. dn.dev/4k-kubernetes Capabilities ● Rapid application deployment ● Scale-to-zero ○ No pod == no memory ● Scale-from-zero ○ Traffic spike starts N pods ● Configurations and Revisions ○ Deployment snapshots ○ Built-in Blue/Green ● Routing and Network configurations
- 41. dn.dev/4k-kubernetes Exploring Knative Eventing ● Eventing Sources ○ Producers of events e.g. Github, Camel, GCP Pub/Sub,CronJob ● Eventing Channels ○ Event forward or persistence layer e.g. Apache Kafka Channel, GCP Pub/Sub , InMemory ● Brokers ○ Event mesh ● Triggers ○ Connecting Broker and Subscriber ○ Filtering Events
- 42. dn.dev/4k-kubernetes Exploring Knative Eventing (contd.,) ● Event Registries ○ Repository of available event types that can be used with Broker ● Event Sink ○ Kubernetes Service ○ Knative Service ○ Channel ○ Broker
- 43. dn.dev/4k-kubernetes CNCF, CloudEvents and Serverless https://cloudevents.io/ https://www.cncf.io/
- 47. dn.dev/4k-kubernetes Apache Kafka ? ● Developed at Linkedin back in 2010, open sourced in 2011 ● Designed to be fast, scalable, durable and available ● Ability to handle huge number of consumers ● Distributed by nature ● Messages are sent to and received from a topic
- 48. dn.dev/4k-kubernetes DevOps challenges - Apache Kafka ● Installation and Configuration is not simple ● Configuring a new cluster: ○ Configuring single brokers ○ Pay attention to conflicting configuration ● Updating a cluster: ○ Updating every single broker ○ Rolling restart of Brokers ○ Rollout and synchronization of Apache Zookeeper
- 50. dn.dev/4k-kubernetes Topic & User operators Zookeeper Cluster Kafka Cluster Cluster Operator Kafka Custom Resource Deployment Apache Kafka on Kubernetes
- 51. dn.dev/4k-kubernetes When to use Apache Kafka ? ● Kafka is a great tool for event-driven architectures ○ Backbone for event driven systems ○ Streaming platform ● Events can have different forms ○ Absolute: Temperature is 21 degrees celsius ○ Deltas: Temperature increased by 1 degree celsius ● Event streams ○ Event sourcing ○ Audit trails ○ Log and metrics analytics
- 55. dn.dev/4k-kubernetes What is Apache Camel •The swiss knife of integration •>10 years of development - still one of the most active Apache projects •Java-based integration framework that can be installed on multiple runtimes: Spring-Boot, JEE, JBoss, Karaf, Standalone, ... •Based on Enterprise Integration Patterns •Supports 250+ components •Uses a powerful DSL •Can integrate anything
- 58. dn.dev/4k-kubernetes Apache Camel-K ? ● Lightweight integration platform to run Apache Camel integrations as Kubernetes applications ● Run integrations as serverless applications ● Ability to integrate with Knative Channels
- 59. dn.dev/4k-kubernetes Development Environment kamel CLI Remote Cloud Integration Definition Camel-K Operator Live Update Developer Workflow
- 62. dn.dev/4k-kubernetes Resources Tutorials •Kubernetes Tutorial - dn.dev/kube-tutorial •Knative Tutorial – dn.dev/knative-tutorial •Istio Tutorial - dn.dev/istio-tutorial •Quarkus Tutorial - dn.dev/quarkus-tutorial •Tekton Tutorial - dn.dev/tekton-tutorial Master Courses - dn.dev/master DevNation Tech Talks - dn.dev/upcoming