2. Agenda
Intro to Splunk for Application Development
Splunk Developer Platform
Web Framework
REST API
SDKs and Tools
Some Sample Apps
Support and Community
2
3. Setting expectations
•
This session does not replace training (not even close)
•
We are not walking through building an App
•
I am not debugging your source code
•
I will show you how to use Splunk from the command line
•
I will show examples of the REST API
•
I will show some sample work
3
6. How Do Developers Use Splunk?
Accelerate
Dev & Test
Integrate with IT
Infrastructure
Build Real-time Big Data
Applications
7. Accelerate
Dev & Test
Every Application Developer Should
Use Splunk
– Get applications ready for production faster
– Find and fix bugs
– Trace transactions in real time
– Build operational intelligence into your apps
without defining a schema
– Use semantic logging for better insight gathering
8. Unlock the power of Splunk for the
entire enterprise
– Search, manage and visualize
Splunk data outside of Splunk
Fully documented and supported
REST API
– Extremely flexible, over 170
endpoints
Fully documented and supported
SDKs for
Python, Java, JavaScript, PHP, Ruby,
and C#
– Support for popular, open
languages
Integrate with IT
Infrastructure
9. Splunk offers a fully-integrated
platform
– Collection, storage, query language, visualization “outof-the-box”
– Real-time insights: clickstream analysis, IT earlywarning systems, security and fraud protection
– Enterprise-grade scale and access control
– Support for popular, open languages
Build Real-time Big Data
Applications
10. Performance Testing for Dev Ops
“We use Splunk to monitor the full software
development cycle - from version control
commits, to continual integration builds, to agile
issue tracking tools, to continual deployment stats.
All of the data combined can be used to illustrate the
health of development efforts in real-time.”
Dan Cundiff
Production Engineer
• Splunk monitors activity from code commit through production
• Target uses Splunk to ensure real-time health and stability of
continuous integration for DevOps agility and responsiveness
10
11. Running Real-time Searches with the Java SDK
“Splunk lets us find and fix
issues on a customer’s DVR
before anyone makes a phone
call.”
Travis Parchman
Operations
•
•
11
Support Reps issue real-time searches from a custom CSR
app to inspect customer DVR health
Proactive maintenance leads to lower support costs for
Comcast and higher customer satisfaction
12. Powering customer-facing apps with Splunk
“The Splunk Python SDK provides us
with a familiar developer environment
to build an application on Splunk that
provide metrics and comparables to our
entire customer base.”
Bill Matthews
CTO
•
•
MSP providing network monitoring and security management
Using the Python SDK to build custom dashboards for
customers powered by data from Splunk
12
13. What You Need to get started
•
•
•
•
•
Splunk
Data
Text Editor or IDE
Documentation on dev.splunk.com
SDKs on dev.splunk.com & https://github.com/splunk
(* They are FREE!)
13
15. The Splunk Platform
Inputs, Apps, Other
Content
Operational Intelligence Platform
UI
Content
SDK
REST API
Core Functions
User and Developer Interfaces
Core Engine
Search Processing Language
Indexing
Collection
15
16. Powerful Platform for Enterprise Developers
Build Splunk Apps
Web
Framework
Extend and Integrate Splunk
Simple XML
SDKs
Data Models
JavaScript
Ruby
Java
JavaScript C#
PHP
Python
Search Extensibility
Django
REST API
16
Modular Inputs
17. Hunk: Powerful Developer Platform on Hadoop
Web
Framework
Simple XML
SDKs
Data Models
JavaScript
Ruby
Java
JavaScript C#
PHP
Python
Search
Extensibility
Django
REST API
Hadoop
Storage
17
19. Splunk Web Framework
Brief History / Overview
• Available as a stand-alone web
•
•
•
•
framework in 5.0
Built into Splunk 6.0
Created for web developers
Pre-built components
Splunk dashboard styles
19
20. Splunk Web Framework
Familiar Technologies
Web
Framework
- Code with JavaScript & Django/Python
- HTML5/CSS/JS Support
- Built on JQuery & Backbone.js
Simple XML /
HTML
Flexible and Extensible
JavaScript
- Create custom layouts
- Integrate visualizations like Sankey
charts, heat maps and bubble charts
- SimpleXML to HTML Conversion
Django
20
21. Get More Familiar with Web Framework
Web Framework App - http://apps.splunk.com/app/1613/
Web Framework Components:
•
•
•
•
•
•
•
Search Managers
Charts
Tables
Forms
Search Manager Controls
Map
Data Template
21
22. Using the Web Framework
Toolkit components
Where to find the components:
– $SPLUNK_HOME/etc/apps/splunk_wftoolkit/django/splunk_wftoolkit/static/spl
unk_wftoolkit/components/
When the toolkit is not installed:
– Include components in your app
– Remember to keep them updated!
When the toolkit is installed:
– Reference components in the toolkit
Each of the following examples also include Example Code
22
28. The REST API (and SDKs)
Search
Visualize
Manage
Create and run
searches from
other applications
Integrate search
results with other
applications using
custom
visualizations
Add/Delete Users
28
29. The Splunk REST API
Exposes an API method for every feature in the product
– Whatever you can do in the UI – you can do through the API
– Run searches
– Manage Splunk configurations
API is RESTful
–
–
–
–
–
Endpoints are served by splunkd
Requests are GET, POST, and DELETE HTTP methods
Responses are Atom XML Feeds
Versioning introduced in Splunk 5.0
Search results can be output in CSV/JSON/XML/raw
29
30. What else about the REST API?
Common HTTP Status Codes
– Returned after all endpoint requests (All your favorites including 404)
Authentication Methods
– HTTP Header
– HTTP Basic
Global pagination and filtering
– Returned lists of objects adhere to a standard interface
Object Sharing and Permissions
– All endpoints that list user objects support object sharing and Access Control List
(ACL)
30
31. Where can you make API requests?
A terminal
Web browser
Any code
Lets see some examples!!!
31
33. Overview of the Splunk SDKs
Currently, Splunk has SDKs for
these languages:
•
•
•
•
•
•
What you can do
• Integrate with 3rd-party tools
• Log directly to Splunk
• Integrate Splunk search results into
Python
Java
JavaScript
PHP
Ruby
C#
your application
• Extract data for archiving
• Build a UI on the web stack of your
choice
• ...and so much more
33
34. What the Splunk SDKs do for you
Handling HTTP access including certs
Authenticating – a session key can be used for subsequent requests
Managing namespaces
Simplifying access to REST endpoints
Building the correct URL for an endpoint
Displaying simplified output for searches
Simplification!
34
40. Blocking, One Shot, Real Time Searches
// Always block until results are ready.
queryArgs.put("exec_mode", "blocking");
Job job = service.getJobs().create(query, queryArgs);
// Execute the oneshot query, which returns the stream (i.e. there is
// no search job created, just a one time search)
InputStream stream = service.oneshotSearch(query, queryArgs);
// Always set real time search mode; No need to wait on a job
queryArgs.put("search_mode", "realtime");
40
45. Anonymous Mood Generator
• Moral and mood towards different projects and tasks based on
Google Forms Survey.
45
46. Closer look
Network device Modular Input
Custom Splunk Command
Not on apps.splunk.com yet, they are beta
– Available on https://github.com/jamesdon
46
49. Where to Go for More Info
•
Tutorials, Code Samples, Getting Started, Downloads
– http://dev.splunk.com/ & http://dev.splunk.com/hunk
•
Support
– https://www.splunk.com/page/submit_issue
•
GitHub
– https://github.com/splunk/
•
Twitter
– https://twitter.com/splunkdev
•
Blog
– http://blogs.splunk.com/dev/
49
The Splunk Development Platform makes it easy to customize and extend Splunk to make the most out of your IT investments. Developers use Splunk in 3 ways
Core Splunk “out of the box” increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in productionIn your org you probably already use Splunk for App managementBut Splunk is a great tool for App Development as well“Code isn’t complete unless it’s Splunk friendly.”Splunk is a great solution for both dev and test, from the unit level to full integration
The Splunk Developer Platform allows you to deliver greater operational agility throughout their organization by making it easy to integrate data from Splunk with other applications. Splunk provides a fully documented and supported REST API with over 170 endpoints that let developers programmatically index, search and visualize data in Splunk from any application. Splunk’s SDKs let developers integrate Splunk data with other applications across the enterprise, from custom-built mobile reporting apps to off-the-shelf CRM solutions, using familiar languages and frameworks.
Build Real-time Data Applications with the SDKs / REST APISplunk is a fully-integrated platform that delivers rapid “time-to-value” to developers. Many of our customers are building robust applications on Splunk today that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at a scale that their businesses demand.This is not limited to everyday customers, as vendors are using Splunk’s analytics engine to power their own Aps.
Target Splunk’s data from Jenkins (build server) to monitor their continuous deployment/dev ops.
Splunk Use Case: Comcast has many different Splunk use cases. Comcast’s X1 Platform Device Manager CSR app uses the Java SDK to issue (one-shot) queries (complex, with multiple sub-queries) to find errors of the last 60 minutes (needs to be realtime – submits the query syntax live). Looking for DVR attempts/failures, Baud attempts/failures, etc. Proactive investigation of customer DVRs - reps need to see if customers are actually using their DVR (I.e. Don't want to reboot a box if customers are watching TV). The goal is to fix the error before anyone makes a phone call (saving on support operations costs and increasing customer satisfaction).Before Splunk, DVR box daemon's were batch polled once a day, which was much less effective identifying errors, leading to higher call volume and lower sat. Business Value: Lower support costs& increased customer satisfaction
IDE - integrated development environment
As the Splunk platform evolves you can see how the Development capabilities have expanded to enable developers to better build on Splunk’s core capabilities. Let’s double click into the Developer platform….Whatdoes this platform look like?The platform consists of 2 layer:A core engine and an interface layerOn top of the platform you can’t run a broad spectrum of content that supports use casesUse cases range from application mgmt. and IT operations, to ES and PCI compliance, to web analyticsThe core engine provides the basic services for real time data input, indexing and search as well alerting, large scale distributed processing and role based accessThe Interface layer consist of the basic UI for search, reporting and visualization– it contains developer interfaces, the REST API and SDKsThe SDKs provide a convenient access to core engine services in a variety of programing language environments. These programmatic interfaces allow you to either:extend Splunkintegrate Splunk with other applicationsbuild completely new applications from scratch that require OI or analytical services that Splunk provides
BUILD SPLUNK APPSThe Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to BUILD interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).EXTEND AND INTEGRATE SPLUNKThe Splunk SDKs include documentation, code samples, resources and tools to make it faster and more efficient to program against the Splunk REST API using constructs and syntax familiar to developers experienced with Java, Python, JavaScript, PHP, Ruby and C#. Developers can easily manage HTTP access, authentication and namespaces in just a few lines of code. Developers can use the Splunk SDKs to: - Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications - Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboards- Build mobile applications with real-time KPI dashboards and alerts powered by Splunk - Log directly to Splunk from remote devices and applications via TCP, UDP and HTTP- Build customer-facing dashboards in your applications powered by user-specific data in Splunk - Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of Splunk- Programmatically extract data from Splunk for long-term data warehousingSplunk Enterprise offers search extensibility through: - Custom Search Commands - Scripted Lookups- Scripted Alerts- Search Macros
Hunk: The Most Powerful Platform for Building Big Data Apps on HadoopHunk is the fully-featured platform that provides rapid exploration, analysis and visualization of your data at rest in Hadoop. It’s based on years of experience building big data products deployed at thousands of Splunk customers and drives dramatic improvements in the speed and simplicity of getting insights from big data in Hadoop. Hunk works with Apache Hadoop or the Hadoop distribution of your choice including first-generation MapReduce and YARN. Hunk also provides a rich developer platform including an integrated Web Framework and Software Development Kits (SDKs) for the world’s most popular development languages. With Hunk, you can build apps powered by data stored in Hadoop Distributed File System (HDFS) as well as integrate data from HDFS into other applications and systems without having to manually program MapReduce jobs.Hunk provides a rich developer environment that enables you to build powerful enterprise Big Data apps that deliver business insights like clickstream analysis, deep customer behavioral modeling and security analysis at enterprise-grade scale using the languages, frameworks and tools you know. Applications built on Hunk can deliver segmented, secure views of data through a highly customizable, flexible interface. The Splunk Web Framework makes building an app on top of Hadoop look and feel like building any modern web application. Developers can also integrate and extend the power of Hunk. The REST API and Software Development Kits (SDKs) for Java, JavaScript, Python, C#, Ruby and PHP, enable you to integrate data and functionality from Hunk into other applications across the enterprise, from custom-built mobile reporting apps to Web Parts in Microsoft SharePoint.Hunk offers ad hoc exploration, analysis and visualization of historical data at rest in Hadoop. Dynamically query data in HDFS or write a custom search script in a few lines of Python without having to cobble together numerous other projects and components or set up MapReduce. Hunk utilizes the Splunk Search Processing Language (SPL™), the industry-leading method to enable interactive data exploration across large, diverse data sets. With Hunk’s schema-on-the-fly, users are not limited or constrained by rigid schemas and can immediately query and interrogate raw data in Hadoop through visual interactions and SPL for deeper analysis. You can also expand the search language to customize commands to perform custom processing or calculations and trigger a shell script or batch file with scripted alerts. You can write custom search commands in Python that take data in standard input (stdin) and output data on standard output (stdout). Once that Python script has been written, all it takes is a simple configuration setting to add the custom search command to the Hunk search pipeline. Additionally, customers with both Splunk Enterprise and Hunk licenses can search across data stored both in Hadoop and in native indexes in Splunk Enterprise – all in the same search.
You can use the new frame work in the latest two versions of Splunk. Splunk as a product is ever evolving.
Splunk is a fully-integrated platform that delivers rapid time-to-value to application developers. Developers can build robust applications on Splunk that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at enterprise-grade scale using the languages, frameworks and tools that they know and love. Applications built on Splunk can deliver segmented, secure data to customers in any UI, powering your core product or service with real-time operational intelligence making Splunk more valuable and more relevant to more users. The Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to build interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).Use Simple XML for fast, lightweight dashboard building as well as add custom styling, behavior and visualizationsUse JavaScript for client-side development Use Python and the Django framework for server-side developmentAdvanced XML?
Splunk has published a Web Framework App to make getting more familiar with it much more simple. Maps are now part of Splunk 6.0Search managers correspond to your Splunk search jobs, saved reports, and post-process searches.Splunk views are used to visualize data, accept form input, and control searches.Tokens and data binding provides a handy mechanism for you to bind values together from searches and views to keep them in sync.Drilldown actions let you customize the drilldown behavior when users click on views.Splunk dashboard styles are used by HTML dashboards for layout, but you can use the dashboard stylesheet in your apps too.DataTemplate view with an embedded template using Django
The Web Framework Toolkit includes a number of new, reusable components based on open-source libraries. These toolkit components are examples that demonstrate how to build your own custom components for your own apps, as well as supported add-ons for others to include in their apps.
Overall the
The Splunk SDKs empower developers to deliver greater operational agility throughout the enterprise by making it easy to integrate data from Splunk with other applications. Splunk provides a fully-documented and supported REST API with nearly 200 endpoints that let developers programmatically index, search and visualize data in Splunk from any application. Splunk’s SDKs, built on that API, make it easy for developers to integrate data from Splunk with other applications across the enterprise, from custom-built mobile reporting apps to off-the-shelf CRM solutions. Splunk offers SDKs for Python, Java, JavaScript, PHP, Ruby and C#. Developers can use the Splunk SDKs with Hunk to: Run searches and retrieve Splunk data from line-of-business systems like Customer Service applications Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboardsBuild customer-facing dashboards in your applications powered by user-specific data in Splunk Manage a Splunk instance, including adding and removing users
IndexLog directly to Splunk (TCP, UDP, HTTP)SearchIncluding saved searchesExtract data from SplunkVisualizeIntegrate search results with third-party reporting tools, portals and other custom applicationsManageAdd/remove users and rolesCreate inputs
2 main categories – Search and Manage
Although you can use the REST API directly, you can also use the Splunk SDKs to interact with Splunk. Essentially, these SDKs are wrappers around the REST API that do a lot of the work for you, such as:
All communication to the Splunk REST API is through the Service object.
In Splunk, every object you create is tied to an owner and an app – you can manage access control via specifying namespaces.
