Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Building Open Source Communities for AWS Serverless Developer Tools

883 vues

Publié le

AWS supports open source. AWS developers have contributed to key projects like Apache MXNet, Gluon API, EKS, and Serverless developer tools. Serverless services like AWS Lambda allow developers to build and run applications without thinking about servers. In 2016, we released an open specification called Serverless Application Model (SAM), a simple configuration file to define Lambda functions and other serverless resources. Since then we have open sourced the underlying implementation of SAM and several other tools to simplify the process of building serverless applications, including SAM Local, a popular CLI tool to run SAM-based applications on a local computer before deploying to the cloud.

In this talk, we touch upon the story of open sourcing the SAM toolset. The talk will deep dive into how an open specification has kindled and nurtured an ecosystem of open source projects comprising of serverless examples, reference architectures, libraries, CLIs, and plugins. We will share some success stories of serverless projects like Chalice that went open and matured to become production-grade. We will also discuss the journey of SAM Local CLI from a weekend project to being open source with 6,000 downloads per month, 2,000+ Github Stars, 30+ contributors, and becoming an integral part of the toolchain. The developer community has helped SAM Local CLI work seamlessly on Windows, support APIs defined in Swagger files, improve unit test coverage, and support a lot of important features. The talk will also cover lessons learned and what worked for us in growing the serverless developer community.

  • Soyez le premier à commenter

Building Open Source Communities for AWS Serverless Developer Tools

  1. 1. Hello I am Sanath Kumar Ramesh. I am a software engineer in AWS Serverless organization where I lead developer tools initiatives. I develop tools to simplify the process of building serverless applications. I built AWS SAM and SAM CLI, open source toolchains to help developers be more productive. 1
  2. 2. A brief background about me.. I started designing computer processors, then built parts of Windows kernel, and graduated up the stack to build web services at startups followed by AWS. In 2016 I joined AWS Lambda team, few months after the service was generally available. I worked on the inner guts of the service but quickly found my passion with developer tools. I have been a huge supporter and evangelist of open source since college. I started the first OSS club in undergrad which is still running successfully. I open sourced the underlying implementation of AWS SAM, built SAM CLI, and work on 100% open source software at AWS. In this talk, I will introduce you to the amazing open source community of serverless developer tools which I have been a part of since 2016. For the longest time, I wasn’t conscious of the scale and potential of this community, until I embarked on my own journey of open source two strategic products at AWS. We will talk about the broader open source AWS community first before diving deep into serverless. 2
  3. 3. Happy Birthday Open Source !! Twenty years! How amazing is that ?? 3
  4. 4. At AWS, we have been a huge believer and supporter of Open Source software. Did you know that some of our foundational services are built on top popular OSS projects - EC2 uses Xen, RDS supports MySQL, EKS supports Kubernetes, to name a few. We go beyond just support and use software, we help build communities. Grow communities: Communities of developers - AWS joined open source foundations like CNCF, OSI, Apache Software Foundation, Linux Foundation etc. We have supported some of our customers like Netflix and Capital One to help build their open source programs. Improve code: We have hunderds of open source projects on our Github repositories, contribute to projects like Kubernetes, Apache MX Net etc. Increase Contributions: Internally, we have several initiatives that make it easier for employees to contribute open source software. For example, using Standard Licenses (Apache or MIT for most part), AWS credits for academic and non-profit projects, Encouraging external communications etc 100s of open source repos on Github: Where we have open sourced everything from SDKs, client side tools, IDE plugins, full blown apps, to core components of some of our services such as s2n. 4
  5. 5. • We are motivated to collaborate by many of our partners and customers • Open Source projects helps us to innovate • Scaling AWS services around open source helps us meet customer needs • Scaling for open data help customers build services they need (Customers! Customers = community. ) 5
  6. 6. 6
  7. 7. 2016 projects, new contributions to apache mxnet Contibution volume is growing every year 7
  8. 8. 2017 projects so far, new ones in orange, notably our first contributions to Kubernetes 8
  9. 9. 9
  10. 10. That was the broader AWS landscape. Let's drill down to the focus of today's talk - Serverless! By show of hands, how many of you have heard of the term Serverless? How many have built serverless apps - hobby, for work, anything? How many of you build & maintain serverless apps in production today? Fantastic! What's all the buzz about serverless? Why do we care? And, Why is Open Source in serverless such a big deal? This is going to be the focus of our talk. 10
  11. 11. No servers to mange, never pay for idle, apps you build scale with usage automatically, and highly available & fault tolerant. AWS Lambda offers a serverless computing platform, API Gateway offers an hosted API service, DynamoDB offers a database etc. For any of you that have built apps on the cloud, serverless offers several system properties that you otherwise have to build yourself. 11
  12. 12. Secret sauce? The things you usually write code for, are now expressed as simple, static, configurations. For example, in a standard cloud application, you would write code for your business logic, API interfaces, and all plumbing between them. Resources like databases, storage, permissions etc are expressed as configurations. I have written the code for user authentication so many times now that I almost decided to quit programming when I wrote user auth for my hobby project recently. In a serverless app, a lot of the undifferentiated heavy lifting is handled by services for you. You write a static configuration file to configure the API endpoints, user auth, workflows etc and the services takes care of running them for you. This means you focus primarily on the business logic and write some plumbing here and there. This also means, all your code-first developer tools such as IDEs, test harnesses, build systems, deployment engines, etc need to support expressing the configuration. 12
  13. 13. This was an opportunity and a whole slew of developer tools emerged in the community. Guess what? Nearly every one of them is open source! Serverless Framework is the biggest and most popular one by far, going by open source community stats. 26k stars on Github, 80k downloads on NPM, > 450 contributors, and has a rich plugin ecosystem. Architect provides a simple DSL to express your entire app with. Paired with their Javascript library, you can write regular NodeJS apps that will be deployed as serverless apps. Apex provides is a simple and opinionated tool to manage your Lambda functions. It can be combined with infrastructure provisioning tools such as Terraform to manage your entire app. Sparta allows you to write Go code that, when deployed, will automatically provision & configure underlying resources for you. Pulumi similarly supports automatic resource provisionng from code, but they support a variety of programing languages, and non-serverless resources too. Claudia.JS is a javascript library to create serverless apps using Nodejs. 13
  14. 14. Zappa is similar to ClaudiaJS but for Python. All of these tools provide varying levels of convenience, target certain programming languages, or specialize in certain tasks. 13
  15. 15. AWS SAM is a language agnostic way of creating, testing, deploying, and managing serverless apps. Shameless plug: I built SAM and the accompanying CLI. We are going to talk a lot about SAM in today's talk. <<talk about other AWS dev tools as much as time fits>> 14
  16. 16. 15
  17. 17. 16
  18. 18. 17
  19. 19. 18
  20. 20. 19
  21. 21. 20
  22. 22. Each of these projects have their own vibrant ecosystem of hundreds of developers working alongside with AWS. How did this all happen? I can’t speak for other projects, but here is a case study of how I created a community around SAM. 21
  23. 23. 22
  24. 24. 23
  25. 25. Shameless Plug SAM is fully open source and includes a companion CLI tool to locally test & run serverless aplications 24
  26. 26. 25
  27. 27. 26
  28. 28. 27
  29. 29. 28
  30. 30. - Open Specification means: No code but just one spec file on Github. Think of this as essentially our docs are on Github. - Excellent traction on Github. Great community of developers for the first few months. 29
  31. 31. Month or two after launch, someone filed an issue asking us to Open Source the implementation. 30
  32. 32. 31
  33. 33. 32
  34. 34. 33
  35. 35. 34
  36. 36. 35
  37. 37. 36
  38. 38. 37
  39. 39. 38
  40. 40. Kubernetes: Amazon EKS Hadoop: Amazon EMR Key difference with SAM: • We didn’t open source EKS, but instead support Kubernetes as part of the EKS service. This is different from SAM where we will open source the entire service • We are the producer of this product and we run it as a service too.. 39
  41. 41. 40
  42. 42. 41
  43. 43. These questions themselves focus on the “software”, but once I answered these questions, it became apparent that I was “building a community” and not just another piece of software. The process of running an open source project naturally drove me to build communities. Goal is to allow the community to build software with full assistance from AWS. Combined the agility of open source community and quality assurance practices of AWS, we believe this model can produce amazing software. 42
  44. 44. When I proposed open source SAM, my manager asked me how many ppl do I want in this team. I said 100! Only a handful of engineers directly come from AWS, but a large majority are from the community. To build this team, we needed open communication where everyone feels connected and a part of same team. Hence Slack with ~1000 ppl including customers, devs, etc. Builds Bonds: • Customers feel like a part of the product, talk directly to product engineers without several layers of support/management to go thru. They feel more empowered to use our product 43
  45. 45. Because development is open: • Builds Trust • Reduces possibility of malicious code • Commercial friendly Apache 2.0 license invites the community to build innovative solutions on top of SAM. Like for example what Stackery did. Community Protects against: • MALICIOUS code • Poor quality code 44
  46. 46. We have a team, we have a product that is open. Next is to keep everyone on same page of what to work on. In an internal team, product & project managers come up with this list. We use Github Labels & Gitub Issues to keep everyone informed of the priority. • Community is on same page on what to work on next • Visibility into top-issues • Provides customers a clear visibility into how we think about this problem. They can obviously provide feedback to change our mind. This is refreshing instead of the opaque “we have this in our backlog somewhere” 45
  47. 47. Running a Service: * We take care of deployments on behalf of the community. At AWS, everything is a service and we have deep knowledge in deploying services. • Automation – all AWS regions • Safe Guards – Beta, Gamma, Auto-rollback, Alarms, Canaries • Bleeding edge vs Stability: We guarantee stability of code (back compatibility etc) by doing deploymsents with several safe guards. Bleeding edge customers can consume source directly • Open communication on Github Releases to mark global availability 46
  48. 48. * Oncall: To make sure Github Issues & PRs don’t’ go stale, we have a oncall rotation where on person is in-charge per week to manage the repository. Obviously any team member can operate the repo at anytime, but having this person ensures the community is always engaged. • Operational Review: After every week, we have an internal meeting to discuss highlights of the oncall. This provides the entire team visibility into product, community health, emergent issues or anything that could change priorities. • We bring our DevOps experience from AWS into an open source product to make sure both the community & the product stays healthy. • Information from each week’s operational review is visible to broader AWS Serverless organization. • We often trade notes with other popular OSS projects like AWS CLI, SDKs to make sure we don’t repeat same mistakes and continue to raise the bar. 47
  49. 49. We support the community by automatically providing them with the best practices we have learnt at AWS over several years. 48
  50. 50. 49
  51. 51. Operational Practices 50
  52. 52. 51
  53. 53. 52