SlideShare a Scribd company logo
1 of 15
Capturing Network Traffic into
Database
Key Words: Sniffer, Network Analyzer,
Wireshark, MySQL, Database, PCAP
to MySQL
How to Store Packets into Database
(for example, MySQL)
• Having packets in database can be
very convenient:
– More performance
– Parallel writing
– Quick analysis
– Data Mining (if you want)
– Long time storage
How???

PCAP
(or real-time
capturing)

XML output

MySQL
What do we need?
• tshark (supplied with WireShark)
• PHP with XML, Xpath support
• MySQL database
My workstation
• All examples here are done in Windows
7, but if you want, you will not need a
lot of time to make them for Linux
• Our task: capture TCP packets (IP from,
IP to, port from, port to, length,
sequence) into database. Example can
be any, for instance, checking for
network scanning.
Distributed
• This can be distributed, no problem,
but you need to use extra network or
filters (otherwise, you will hang your
system: 1 sniffed packet sent make 1
more, and so on).
Distributed
Remote SQL Server

WorkStation

WorkStation
WorkStation
Getting traffic XML format
• tshark -r "D:test.pcap" -T pdml > D:test_T.xml
– Converting pcap into XML

Or
• tshark -T pdml | you_application.exe
– Real-Time
Output XML example
Warning
• Such converting to XML consume a lot of
space (50x)! PCAP file from 200 Kb grew into
10 Mb XML!!!
• In this case you might find useful to divide one
big pcap file into several of smaller size
• Also filtering is good idea, so you can throw
out fields useless for you.
XML output file structure
• It is very simple (I crossed out trivial parts, so real lines are
bigger):
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?>
<pdml >
<packet>
Packet info
</packet>
<packet>
Packet info
</packet>
And so on.
XML structure – packets (Example,
DNS query)
<packet>
<proto name="geninfo" pos="0" showname="General information" size="73">
</proto>
<proto name="frame" showname="Frame 1: 73 bytes on wire (584 bits), 73 bytes
captured (584 bits)" size="73" pos="0">
</proto>
<proto name="eth">
</proto>
<proto name="ip" >
</proto>
<proto name="udp" showname="User Datagram Protocol, Src Port: 58150 (58150),
Dst Port: domain (53)" size="8" pos="34">
</proto>
<proto name="dns" showname="Domain Name System (query)" size="31" pos="42">
</proto>
</packet>
(Child elements and attributes of proto are not shown here)
XML to MySQL
• You can use LOTS of options: C++/Java,etc.
• I used SimpleXML and XPath with PHP:
$file = "test_T.xml";
$my_file = simplexml_load_file($file );
foreach ($my_file >xpath('//packet') as $packet)
{
$packet_type = $packet->proto[4];
echo $packet_type['name']; //protocol
}
And putting into database
function LoadToDataBase($con)
{
$stmt =$con->prepare("INSERT INTO tcp (capture_order, from_ip,
to_ip, from_port, to_port, tcp_length, tcp_stream, tcp_stream_text,
tcp_sequence_dec) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)");

$stmt->bind_param('sssiiiisi', $this->capture_order,$this->from_ip,
$this->to_ip,
$this->from_port,$this->to_port,
$this->tcp_length,$this->tcp_stream, $this->tcp_stream_text,
$this->tcp_sequence_dec);
$stmt->execute();
}
Here $con is open connection to mysql, and all this vars I got in cycle.
Please, refer to full code.
Thank you
• I hope you find this useful.

• Full code is available at
http://tigrantsat.me/randd/pcaptomysql/

More Related Content

What's hot

Lecture: Question Answering
Lecture: Question AnsweringLecture: Question Answering
Lecture: Question AnsweringMarina Santini
 
NLP using transformers
NLP using transformers NLP using transformers
NLP using transformers Arvind Devaraj
 
Introduction To RDF and RDFS
Introduction To RDF and RDFSIntroduction To RDF and RDFS
Introduction To RDF and RDFSNilesh Wagmare
 
CS571: Language Models
CS571: Language ModelsCS571: Language Models
CS571: Language ModelsJinho Choi
 
Natural language processing and transformer models
Natural language processing and transformer modelsNatural language processing and transformer models
Natural language processing and transformer modelsDing Li
 
Text classification presentation
Text classification presentationText classification presentation
Text classification presentationMarijn van Zelst
 
Unsupervised Topic Modeling with BERTopic for Coarse and Fine-Grained News Cl...
Unsupervised Topic Modeling with BERTopic for Coarse and Fine-Grained News Cl...Unsupervised Topic Modeling with BERTopic for Coarse and Fine-Grained News Cl...
Unsupervised Topic Modeling with BERTopic for Coarse and Fine-Grained News Cl...MODUL Technology GmbH
 
Attention in Deep Learning
Attention in Deep LearningAttention in Deep Learning
Attention in Deep Learning健程 杨
 
Deep learning for NLP and Transformer
 Deep learning for NLP  and Transformer Deep learning for NLP  and Transformer
Deep learning for NLP and TransformerArvind Devaraj
 
Natural Language Processing
Natural Language ProcessingNatural Language Processing
Natural Language ProcessingVeenaSKumar2
 
Natural Language Processing
Natural Language Processing Natural Language Processing
Natural Language Processing Adarsh Saxena
 
Machine Learning Models in Production
Machine Learning Models in ProductionMachine Learning Models in Production
Machine Learning Models in ProductionDataWorks Summit
 
Data engineering zoomcamp introduction
Data engineering zoomcamp  introductionData engineering zoomcamp  introduction
Data engineering zoomcamp introductionAlexey Grigorev
 
The Business Case for Semantic Web Ontology & Knowledge Graph
The Business Case for Semantic Web Ontology & Knowledge GraphThe Business Case for Semantic Web Ontology & Knowledge Graph
The Business Case for Semantic Web Ontology & Knowledge GraphCambridge Semantics
 
The Semantic Web #9 - Web Ontology Language (OWL)
The Semantic Web #9 - Web Ontology Language (OWL)The Semantic Web #9 - Web Ontology Language (OWL)
The Semantic Web #9 - Web Ontology Language (OWL)Myungjin Lee
 
Neural Text Embeddings for Information Retrieval (WSDM 2017)
Neural Text Embeddings for Information Retrieval (WSDM 2017)Neural Text Embeddings for Information Retrieval (WSDM 2017)
Neural Text Embeddings for Information Retrieval (WSDM 2017)Bhaskar Mitra
 
GAN in medical imaging
GAN in medical imagingGAN in medical imaging
GAN in medical imagingCheng-Bin Jin
 

What's hot (20)

Lecture: Question Answering
Lecture: Question AnsweringLecture: Question Answering
Lecture: Question Answering
 
NLP using transformers
NLP using transformers NLP using transformers
NLP using transformers
 
Introduction To RDF and RDFS
Introduction To RDF and RDFSIntroduction To RDF and RDFS
Introduction To RDF and RDFS
 
CS571: Language Models
CS571: Language ModelsCS571: Language Models
CS571: Language Models
 
Natural language processing and transformer models
Natural language processing and transformer modelsNatural language processing and transformer models
Natural language processing and transformer models
 
Text classification presentation
Text classification presentationText classification presentation
Text classification presentation
 
Unsupervised Topic Modeling with BERTopic for Coarse and Fine-Grained News Cl...
Unsupervised Topic Modeling with BERTopic for Coarse and Fine-Grained News Cl...Unsupervised Topic Modeling with BERTopic for Coarse and Fine-Grained News Cl...
Unsupervised Topic Modeling with BERTopic for Coarse and Fine-Grained News Cl...
 
Attention in Deep Learning
Attention in Deep LearningAttention in Deep Learning
Attention in Deep Learning
 
Deep learning for NLP and Transformer
 Deep learning for NLP  and Transformer Deep learning for NLP  and Transformer
Deep learning for NLP and Transformer
 
Natural Language Processing
Natural Language ProcessingNatural Language Processing
Natural Language Processing
 
Natural Language Processing
Natural Language Processing Natural Language Processing
Natural Language Processing
 
Machine Learning Models in Production
Machine Learning Models in ProductionMachine Learning Models in Production
Machine Learning Models in Production
 
Prolog
PrologProlog
Prolog
 
Data engineering zoomcamp introduction
Data engineering zoomcamp  introductionData engineering zoomcamp  introduction
Data engineering zoomcamp introduction
 
The Business Case for Semantic Web Ontology & Knowledge Graph
The Business Case for Semantic Web Ontology & Knowledge GraphThe Business Case for Semantic Web Ontology & Knowledge Graph
The Business Case for Semantic Web Ontology & Knowledge Graph
 
The Semantic Web #9 - Web Ontology Language (OWL)
The Semantic Web #9 - Web Ontology Language (OWL)The Semantic Web #9 - Web Ontology Language (OWL)
The Semantic Web #9 - Web Ontology Language (OWL)
 
RDF and OWL
RDF and OWLRDF and OWL
RDF and OWL
 
Neural Text Embeddings for Information Retrieval (WSDM 2017)
Neural Text Embeddings for Information Retrieval (WSDM 2017)Neural Text Embeddings for Information Retrieval (WSDM 2017)
Neural Text Embeddings for Information Retrieval (WSDM 2017)
 
Word2Vec
Word2VecWord2Vec
Word2Vec
 
GAN in medical imaging
GAN in medical imagingGAN in medical imaging
GAN in medical imaging
 

Viewers also liked

Convert Wireshark PCAP Files to Sequence Diagrams
Convert Wireshark PCAP Files to Sequence DiagramsConvert Wireshark PCAP Files to Sequence Diagrams
Convert Wireshark PCAP Files to Sequence DiagramsEventHelix.com Inc.
 
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...EAE Business School
 
Open source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysisOpen source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysisGTKlondike
 
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013Sergey Petrunya
 
CapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet InspectionCapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet InspectionChris Harrington
 
Wireshar training
Wireshar trainingWireshar training
Wireshar trainingLuke Luo
 
Authentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slidesAuthentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slidesrahul kundu
 
Radius server,PAP and CHAP Protocols
Radius server,PAP and CHAP ProtocolsRadius server,PAP and CHAP Protocols
Radius server,PAP and CHAP ProtocolsDhananjay Aloorkar
 
Implementing Cisco AAA
Implementing Cisco AAAImplementing Cisco AAA
Implementing Cisco AAAdkaya
 
What is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioningWhat is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioningMahindra Comviva
 
Wireshark
WiresharkWireshark
Wiresharkbtohara
 
Radius Protocol
Radius ProtocolRadius Protocol
Radius ProtocolNetwax Lab
 

Viewers also liked (20)

Convert Wireshark PCAP Files to Sequence Diagrams
Convert Wireshark PCAP Files to Sequence DiagramsConvert Wireshark PCAP Files to Sequence Diagrams
Convert Wireshark PCAP Files to Sequence Diagrams
 
Traffic types in internet
Traffic types in internetTraffic types in internet
Traffic types in internet
 
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
 
Open source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysisOpen source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysis
 
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
 
CapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet InspectionCapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet Inspection
 
AfterGlow
AfterGlowAfterGlow
AfterGlow
 
Wireshar training
Wireshar trainingWireshar training
Wireshar training
 
AAA in a nutshell
AAA in a nutshellAAA in a nutshell
AAA in a nutshell
 
Authentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slidesAuthentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slides
 
Radius server,PAP and CHAP Protocols
Radius server,PAP and CHAP ProtocolsRadius server,PAP and CHAP Protocols
Radius server,PAP and CHAP Protocols
 
RADIUS
RADIUSRADIUS
RADIUS
 
Quality of Service
Quality of ServiceQuality of Service
Quality of Service
 
Implementing Cisco AAA
Implementing Cisco AAAImplementing Cisco AAA
Implementing Cisco AAA
 
What is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioningWhat is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioning
 
AAA Protocol
AAA ProtocolAAA Protocol
AAA Protocol
 
Wireshark
WiresharkWireshark
Wireshark
 
Introduction to Diameter Protocol - Part1
Introduction to Diameter Protocol - Part1Introduction to Diameter Protocol - Part1
Introduction to Diameter Protocol - Part1
 
Radius Protocol
Radius ProtocolRadius Protocol
Radius Protocol
 
Wireshark
WiresharkWireshark
Wireshark
 

Similar to Capturing Network Traffic into Database

Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopProject Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopDatabricks
 
High Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal CloudHigh Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal CloudMongoDB
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network ProcessingRyousei Takano
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHungWei Chiu
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaPrajal Kulkarni
 
Ga4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger instituteGa4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger instituteMatt Massie
 
Explore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataExplore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataData Con LA
 
Project Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalProject Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalDatabricks
 
Event Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data ProcessorsEvent Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data ProcessorsInvenire Aude
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...DataWorks Summit/Hadoop Summit
 
osdi20-slides_zhao.pptx
osdi20-slides_zhao.pptxosdi20-slides_zhao.pptx
osdi20-slides_zhao.pptxCive1971
 
Hardware Provisioning
Hardware ProvisioningHardware Provisioning
Hardware ProvisioningMongoDB
 
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...confluent
 
DPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettDPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettJim St. Leger
 
Practical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-onPractical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-onQaisSaifQassim
 
seed block algorithm
seed block algorithmseed block algorithm
seed block algorithmDipak Badhe
 
Multitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEMultitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEkawamuray
 
Python (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network AutomationPython (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network AutomationRick Sherman
 
Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Matthias Niehoff
 

Similar to Capturing Network Traffic into Database (20)

Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopProject Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
 
High Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal CloudHigh Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal Cloud
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network Processing
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User Group
 
Jumbo Mumbo in OpenStack
Jumbo Mumbo in OpenStackJumbo Mumbo in OpenStack
Jumbo Mumbo in OpenStack
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and Kibana
 
Ga4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger instituteGa4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger institute
 
Explore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataExplore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and Snappydata
 
Project Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalProject Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare Metal
 
Event Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data ProcessorsEvent Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data Processors
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
 
osdi20-slides_zhao.pptx
osdi20-slides_zhao.pptxosdi20-slides_zhao.pptx
osdi20-slides_zhao.pptx
 
Hardware Provisioning
Hardware ProvisioningHardware Provisioning
Hardware Provisioning
 
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
 
DPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettDPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles Shiflett
 
Practical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-onPractical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-on
 
seed block algorithm
seed block algorithmseed block algorithm
seed block algorithm
 
Multitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEMultitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINE
 
Python (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network AutomationPython (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network Automation
 
Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra
 

Recently uploaded

How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfPrecisely
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionDilum Bandara
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 

Recently uploaded (20)

How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 

Capturing Network Traffic into Database

  • 1. Capturing Network Traffic into Database Key Words: Sniffer, Network Analyzer, Wireshark, MySQL, Database, PCAP to MySQL
  • 2. How to Store Packets into Database (for example, MySQL) • Having packets in database can be very convenient: – More performance – Parallel writing – Quick analysis – Data Mining (if you want) – Long time storage
  • 4. What do we need? • tshark (supplied with WireShark) • PHP with XML, Xpath support • MySQL database
  • 5. My workstation • All examples here are done in Windows 7, but if you want, you will not need a lot of time to make them for Linux • Our task: capture TCP packets (IP from, IP to, port from, port to, length, sequence) into database. Example can be any, for instance, checking for network scanning.
  • 6. Distributed • This can be distributed, no problem, but you need to use extra network or filters (otherwise, you will hang your system: 1 sniffed packet sent make 1 more, and so on).
  • 8. Getting traffic XML format • tshark -r "D:test.pcap" -T pdml > D:test_T.xml – Converting pcap into XML Or • tshark -T pdml | you_application.exe – Real-Time
  • 10. Warning • Such converting to XML consume a lot of space (50x)! PCAP file from 200 Kb grew into 10 Mb XML!!! • In this case you might find useful to divide one big pcap file into several of smaller size • Also filtering is good idea, so you can throw out fields useless for you.
  • 11. XML output file structure • It is very simple (I crossed out trivial parts, so real lines are bigger): <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?> <pdml > <packet> Packet info </packet> <packet> Packet info </packet> And so on.
  • 12. XML structure – packets (Example, DNS query) <packet> <proto name="geninfo" pos="0" showname="General information" size="73"> </proto> <proto name="frame" showname="Frame 1: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)" size="73" pos="0"> </proto> <proto name="eth"> </proto> <proto name="ip" > </proto> <proto name="udp" showname="User Datagram Protocol, Src Port: 58150 (58150), Dst Port: domain (53)" size="8" pos="34"> </proto> <proto name="dns" showname="Domain Name System (query)" size="31" pos="42"> </proto> </packet> (Child elements and attributes of proto are not shown here)
  • 13. XML to MySQL • You can use LOTS of options: C++/Java,etc. • I used SimpleXML and XPath with PHP: $file = "test_T.xml"; $my_file = simplexml_load_file($file ); foreach ($my_file >xpath('//packet') as $packet) { $packet_type = $packet->proto[4]; echo $packet_type['name']; //protocol }
  • 14. And putting into database function LoadToDataBase($con) { $stmt =$con->prepare("INSERT INTO tcp (capture_order, from_ip, to_ip, from_port, to_port, tcp_length, tcp_stream, tcp_stream_text, tcp_sequence_dec) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)"); $stmt->bind_param('sssiiiisi', $this->capture_order,$this->from_ip, $this->to_ip, $this->from_port,$this->to_port, $this->tcp_length,$this->tcp_stream, $this->tcp_stream_text, $this->tcp_sequence_dec); $stmt->execute(); } Here $con is open connection to mysql, and all this vars I got in cycle. Please, refer to full code.
  • 15. Thank you • I hope you find this useful. • Full code is available at http://tigrantsat.me/randd/pcaptomysql/