Hear how Cisco runs ECE in production on Cisco UCS Integrated Infrastructure and learn about the configurations that help them index 3 billion documents each day at a rate of ~400K docs per second using Apache Spark.

1. 1
Elastic Cloud Enterprise @ Cisco
Sujith Joseph
November 13 2018

2. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
14+ years of Enterprise Search
experience
Enterprise Search Architect
Co-founder
About me..
Enterprise Search & Analytics

3. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
What I will cover today
Deployment
Architecture
Use CasesLessons
Learned

4. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Evolution of Enterprise
Search

5. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Enterprise Search Vendor Landscape - Early 2000’s

6. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco.com Search, Intranet
Search &
Single Enterprise Search
Platform across IT
Centralized Team for Search
Services
100+ search applications
Enterprise Search
Platform @ Cisco IT

7. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Enterprise Search Landscape: Past 8 -10 years

8. 8
4
350M
100M
45M
350,000,000
Downloads
2016 20182015

9. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Elasticsearch @ Cisco IT
Application
Data Search
Log
Analytics
Metrics &
Operational
Analytics
Document
Search

10. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Problems with Elasticsearch Silos
Lack of elastic skillset
For optimal configurations,
management and support
Duplicate Elasticsearch
Operations Work
Performance issues
Bad Infrastructure choices
Search Engine Consolidation
Requests from Management
Stability Issues
Sizing is an afterthought

11. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Why Elasticsearch as a Managed Service? Vision
On-demand provisioning
Secure API
Ease of provisioning
Cloud Experience
Scale up/down
Support for multiple versions
Centralized maintenance
Logging & Monitoring
Resource Isolation
Easy upgrades and re-sizing
Without Downtime

12. Elasticsearch as a Service - Solution Options

13. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-tenant model
Noisy Neighbor problem
IO Rate throttling @ 10K
1
Dedicated VM infrastructure
per Elasticsearch cluster
Low IOPS (15K) for Storage

14. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Persistent Volumes &
StatefulSets
K8s on VMs
2
Works well with single cluster
Elasticsearch Security
features
Additional Development,
Teams needed for managed
service
Designed for general purpose
applications – Low IOPS (25K)
Noisy Neighbor problem

15. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Data Sources - Internal
Applications in Cisco Data Centers
3
Cisco Restricted Content
Cisco Internal Users

16. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Superior Performance
High IOPS (600K – 800K ) - JBOD
4
Docker on Bare Metals
Elastic Security features
Fast Storage (SSD / NVME)
No custom development
needed
Met all of our requirements
Avoided Noisy Neighbor problem

17. 17
Elasticsearch as a
Managed Service
Consultation with Search Team
Sizing & Data Modeling
Increased efficiency for
Application Developers
Better performance & UX for
Applications
24x7 Elastic Support
Consultation
Elastic Security & Machine Learning
features
Cloud experience
Easy resize, cluster & ECE upgrades
without downtimes
ECE - Benefits

18. 18
Fast provisioning of Elasticsearch cluster
helped us fast-track our application
development and rollout in just 2 weeks. We
didn’t have to worry about the installation,
upgrade and management of the
Elasticsearch cluster.
Cisco Internal Elastic Cloud Enterprise User

19. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Business Value for ECE
Increased Innovation
Increased Agility
Faster Realization of
Business Goals
Operational Efficiency

20. 20
Elastic Cloud Enterprise
Deployment Architecture

21. 21
Documents
indexed daily
400,000+ docs per
second indexing
Elastic Cloud Enterprise Scale
Elasticsearch
Clusters
3 availability zones
8 – 64 GB RAM
nodes
Elasticsearch
nodes
4 – 90 nodes
3B 24 270

22. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Configurations
v1.1
v1.11.2
7.2
Bare Metal Servers
Virtual Machines for non-
allocator roles
Load Balancer with Wildcard
SSL
Wildcard DNS

23. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Infrastructure

24. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
INFRASTRUCTURE DC1 DC2
COUNT 3 4
Hardware UCS C Series 240 M4 UCS C Series 240 M5
CPU Cores 72 80
RAM 1.5 TB 1.5 TB
STORAGE 8 * 800 GB SSD (Enterprise
Performance SSD – 3x endurance)
2 * 3.2 TB NVME
(10x endurance)
File System XFS (JBOD) XFS (JBOD)
Infrastructure Specifications
Cisco UCS C240 M5
(for ECE Allocator Roles)

25. 25
ECE Experience in
Production & Learnings

26. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Running in Production for a
year and a half
Highly Stable
~5 minute provisioning vs
2 months (2 years ago)
Our ECE Experience

27. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Central Logging & Monitoring
clusters very useful
Super easy upgrades
High number of Small
Clusters
Our ECE Experience

28. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Upgrade from 512 GB to 1.5
TB for Allocator RAM caused
performance drops
ECE 1.1 – Java Transport
Client not Supported for
elasticsearch 6.x
Minimum replication
factor as 1 for indices
with 2 zones (JBOD)
Learnings
Increased CPU override factor for fix
Use Java High Level REST Client

29. 29
Friends Don't Let Friends Run
Docker on Loopback in
Production
Jason Brooks, Red Hat
Used Docker on Loopback
initially
Changed to overlay as
storage driver [XFS with
ftype=1]

30. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Validate Elastic Data Model
Design for teams before Go-
Live
Provide consultation for sizing
Perform Client Code Review
before Go-Live
Best Practices

31. 31
Elasticsearch Usecases

32. Cisco.com Product Search & Comparison

33. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

34. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

35. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

36. Search Query Understanding

37. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Understand searcher’s intent
Predict task via ML models
Powered by ML Classification
Algorithms & Elasticsearch
Query Rewriting
Used by Cisco.com Search
Instant Answers
Boost & Filter Search Results
What is Query
Understanding?
By intent & entities

38. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Conference Room Booking
End of life
Query Intent Classification
“has end of life been announced for the
catalyst 3750 switch?”
“Book inspire conference room from 2 PM to 4
PM this Wednesday”
IntentSearch Query
Software Download“anyconnect-macosx-i386-2.5.3055-k9.pkg”

39. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
product: catalyst 3750 switch
serial number:
bac10abcd113k8n8
Query Entity Recognition
“Is bac10abcd113k8n8 covered on a service
contract”
“has end of life been announced for the
catalyst 3750 switch”
Entity (key: value)Search Query
product id: ws-c2960-8tc-l“eos ws-c2960-8tc-l”

40. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
cat 3850 catalyst 3850 switch
Query Entity Resolution (powered by Elasticsearch)
“has end of life been announced for
cat 3850”
Entity Resolved EntitySearch Query

41. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Machine Learning Algorithms / Frameworks
MEMM (Maximum-entropy Markov Model) &
Logistic Regression
CRF (Stanford NLP)
CRF + BiLSTM with Character / Word embeddings
Google BERT (POC)

42. SalesConnect Recommendations

43. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
What is SalesConnect?
Personalized
Recommendations
A Z
2ND MOST USED TOOL BY CISCO SELLERS
SINGLE SOURCE OF TRUTH &
ONE STOP SHOP
FOR SALES CONTENT
SALES & PARTNER FOCUSED
CONTENT
Products, Solutions, Services,
Competitive, Leadership, Cisco
Partnerships, Industries, Partner
Programs, Training, Learning Maps, + Much
More!
USING USER’S JOB ROLE
& TOPIC OF INTEREST
FOR SELLERS & PARTNERS

44. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

45. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

46. Sales Compensation Reporting (Sales Visibility)

47. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Problem Statement
Compensation Calculator for
Cisco Sellers
Single Source of Truth
for Data in
Slow responses for reports
(30 seconds)
Timeouts
Issues

48. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
What we tried..
Elastic on Virtual Machines
(30K docs/sec with SAN)
High Query Latency (>10
seconds)
Parent Child Joins with 8
levels of sub-aggregations

49. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
What we finally did..
Elastic on ECE
<800 ms latency for Report
generation
3 billion documents indexed
in 4 hours

50. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
300K - 600K docs/
sec ingestion

51. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Elasticsearch
nodes
3 availability Zones
Sales Visibility - Elasticsearch Sizing
GB RAM
Nodes
16 indices
TB
Index size
296 primary shards
90 16 6

52. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Count - 150
• Cores - 2
• RAM – 12 GB
• Spark Jobs - 14
• Entries – 24,0000
• Size – 15 MB
• indices.memory.index_bu
ffer_size: 20%
• http.max_content_length:
1024mb
Executor ES Spark Batch Size Elastic Configuration
Apache Spark & Elastic Configuration

53. Anomaly Detection using Elasticsearch ML

54. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Anomaly Detection in Data Center Sensor Data

55. Prior Art Search for USPTO Examiners

56. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Problem Statement
US Patent & Trademark Office
Should stop patents from being issued
on old or obvious technology
No Central Archive & Search
for Prior Arts
Ensures No Prior Art exists during
patent examination
Evidences, not just in existing patents,
but in form of public documents &
manuals
Strong Evidence that invention is not
new

57. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
www.priorartarchive.org

58. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Typical User Search

59. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
USPTO Examiner
Search
BRS/Search
(Bibliographic Retrieval Service)

60. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
BRS Syntax
Proximity Search
Fielded Query
Groupings of Boolean clauses of
operands using ( & )
Boolean Search AND, OR
SAME: match within the same paragraph
WITH: match within the same sentence
ADJ: ordered phrase search
NEAR: phrase search (any order)

61. 61
USPTO Search

62. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
USPTO Search strings
• ((connection or session or socket) near2 (error or failure or (time$4 near3 out) or fail$4)) and ((connection or
session or socket) near2 (reclaim$5 or (re adj claim$5) or recover$4 or recycl$5))
• ((failover$3 with (backup$1 or redundan$4)) with (state$1 or health$1 or load$1 or fail$3)) and (virtual adj3
router$1 adj3 redundan$4)
• (TEST$4 SAME EXECUT$4 WITH TIME SAME ORDERING AND
@AD<="20111109").#USOC,DWPI,TDBD,EPAB,JPAB,FPRS,USPT,PGPB
• split$4 with (DNS or domain adj name adj serv$3) with (VPN or virtual adj private adj network or tunnel$4)
• date: [1997-01-01 TO 1999-12-01]

63. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

64. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Prior Art Search Architecture
Messaging
Queue
FSCrawler
Elasticsearch- 5.6
Master Nodes (3)
Data Nodes (3)
Archived Documents /
Prior Art
Corporate Content
Owners
SFTP/API
Archived Content Repository
Ingestion, Audit Microservices
Hosted by
CPC Patent Classification Code Partial Updates
S3 url, Metadata Documents
www.priorartarchive.org
Prior Art Search Portal
elasticsearch
Service
BRS Query Parser
Query

65. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Other Use cases
Customer 360
Search Autosuggestions
Categorized Typeaheads in
Cisco.com Search, Cisco Intranet
Search, Directory Search
Document Search
Application Data Search
Search Analytics
Cisco.com Product
Recommenations
Upsell/Cross-sell
Cisco.com
Personalization

66. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Operational Efficiency with
ECE for Managed Service for
Elastic
Enhancements to existing
Applications, Core of an
Application
Good for Industry
Stopping Patent Trolling
What we achieved..

67. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Questions?

68. Cisco.com Product Recommendations

69. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Product Recommendation on Cisco.com Site Search
Recommending Upsell/Cross sell

70. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Product
Employee_experience
Domain Classification - Examples
“Book inspire conference room?”
“end of service ws-c2960-8tc-l”
DomainSearch Query

71. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Start time : 10:00 AM
End Time : 10:30 AM
Query Role Classification
“book hope conference room next
wednesday from 10 am to 10:30 am”
RangesSearch Query