With the advent of Big Data in the Threat Analytics space needs emerge to perform near real-time (NRT) threat detection and automated interpretation that speed counter measures and remediation. AT&T Chief Security Organization (CSO) has developed an enterprise architecture that includes near real-time outlier processes necessary to protect its network from cyber threats using the Hadoop ecosystem. One enterprise challenge that CSO has faced is summarized in the statement by Brian Rexroad, Executive Director of Technology and Security: "I feel there is too much emphasis is on "detecting". Significantly more emphasis is needed in automated extraction of related information/activity and interpretation of that information." Therefore; CSO Engineering team developed the Stratum™ architecture that includes many open source and commercial products facilitating the rapid development and operationalization of outliner detectors and interpreters. Extensive use of NRT data ingestion, enrichment, organization and random access storage patterns, make these capabilities possible on top of a Hadoop based ecosystem. The Stratum™ architecture offers the CSO the ability to minimize the time and effects of many cyber threats. Using Big Data technologies for cyber threat analysis is becoming quite common, but the need for outlier detection and interpretation is crucial for enterprise protection.
Work real quick through agenda
Just set the stage for an Hadoop based threat analytics platform that has NRT capabilities
Set the stage for how a typical network in this industry and how much work there is for securing it.
Presents an industry problem, not an AT&T problem
Address the outside threat to the internal operation of our industry
Amount of traffic related to reflect based DoS attackers. Illustrates activity on the internet not the attacks against the AT&T perimeter.
Hack-ma-geddon
Columbia government
Spam Hause
Syria <- New York Times
Target lost 40M credit/debit cards
Our TAP has evolved a lot over the last few year as we’ve moved into an Hadoop base architecture. I will briefly describe the roadmap.
Proprietary technology and lack of extensibility are killers
Past was SIEM dependent, based on large RDBMS and exclusively dependent on human detection and interpretation. Largely a data reduction system. Industry solution of yesterday.
The challenge is the cognitive intersection with automation.
An environment of innovation. Goal is to automate the security analysis process which are largely cognitive. Granted this is a different use of Hadoop rather than single use data. Its continual ingestion, NRT detections, alerting, etc. Not always a clear problem statement.
Spend some time developing the human dependency and cognitive processes
Takes a lot of data
Left to right, we move all the data through various processing platforms into an Hadoop base system for raw log management, data org, management, access, analysis and finally to visualization and reporting.