My keynote at the Brazilian Security Symposium (SBSeg), as part of the Computer Forensics Workshop (WFC), talking about fileless malware, the challenges for antivirus detection, and new detection strategies. I present the prototype of a hardware AV with integrated signature matching to decrease the performance penalty imposed by software-only AVs.

1. Introduction Proposed Solution Evaluation Conclusions
Near-memory & In-Memory
Detection of Fileless Malware
Marcus Botacin1
1Texas A&M University (TAMU)
botacin@tamu.edu
SBSEG 2023
Near-memory & In-Memory Detection of Fileless Malware 1 / 52 SBSEG’23

2. Introduction Proposed Solution Evaluation Conclusions
Agenda
1 Introduction
2 Proposed Solution
3 Evaluation
4 Conclusions
Near-memory & In-Memory Detection of Fileless Malware 2 / 52 SBSEG’23

3. Introduction Proposed Solution Evaluation Conclusions
Agenda
1 Introduction
2 Proposed Solution
3 Evaluation
4 Conclusions
Near-memory & In-Memory Detection of Fileless Malware 3 / 52 SBSEG’23

4. Introduction Proposed Solution Evaluation Conclusions
0x0. What is the most concerning type of
malware these days?
Near-memory & In-Memory Detection of Fileless Malware 4 / 52 SBSEG’23

5. Introduction Proposed Solution Evaluation Conclusions
Fileless malware on the news
Figure: Source: https://www.wired.com/2017/02/
say-hello-super-stealthy-malware-thats-going-mainstream/
Figure: Source: https://www.cyberscoop.com/
kaspersky-fileless-malware-memory-attribution-detection/
Near-memory & In-Memory Detection of Fileless Malware 5 / 52 SBSEG’23

6. Introduction Proposed Solution Evaluation Conclusions
0x1. How do fileless malware work?
Near-memory & In-Memory Detection of Fileless Malware 6 / 52 SBSEG’23

7. Introduction Proposed Solution Evaluation Conclusions
Fileless malware infection chain
Figure: Source: https://www.trellix.com/en-us/security-awareness/ransomware/
what-is-fileless-malware.html
Near-memory & In-Memory Detection of Fileless Malware 7 / 52 SBSEG’23

8. Introduction Proposed Solution Evaluation Conclusions
0x2. How hard is to go fileless?
Near-memory & In-Memory Detection of Fileless Malware 8 / 52 SBSEG’23

9. Introduction Proposed Solution Evaluation Conclusions
Fileless malware generation tooks
Figure: Source: https://github.com/nnsee/fileless-elf-exec
Near-memory & In-Memory Detection of Fileless Malware 9 / 52 SBSEG’23

10. Introduction Proposed Solution Evaluation Conclusions
0x3. Is this a real threat?
Near-memory & In-Memory Detection of Fileless Malware 10 / 52 SBSEG’23

11. Introduction Proposed Solution Evaluation Conclusions
Fileless malware in the wild
Figure: Source: https://www.wiz.io/blog/
pyloose-first-python-based-fileless-attack-on-cloud-workloads/
Near-memory & In-Memory Detection of Fileless Malware 11 / 52 SBSEG’23

12. Introduction Proposed Solution Evaluation Conclusions
0x4. Are current AVs ready for that?
Near-memory & In-Memory Detection of Fileless Malware 12 / 52 SBSEG’23

13. Introduction Proposed Solution Evaluation Conclusions
A Drawback for Current Security Solutions
Figure: Default policy is not to scan memory.
Near-memory & In-Memory Detection of Fileless Malware 13 / 52 SBSEG’23

14. Introduction Proposed Solution Evaluation Conclusions
0x5. Why not to scan all the time?
Near-memory & In-Memory Detection of Fileless Malware 14 / 52 SBSEG’23

15. Introduction Proposed Solution Evaluation Conclusions
The Cost of Scanning Memory
0
50
100
150
200
250
300
perl namd Bzip milc mfc
Execution
Time
(s)
Benchmark
AV scanning overhead
Scan
Baseline
Figure: In-memory AV scans worst-case and best-case performance penalties.
Near-memory & In-Memory Detection of Fileless Malware 15 / 52 SBSEG’23

16. Introduction Proposed Solution Evaluation Conclusions
0x6. Where does this overhead come from?
Near-memory & In-Memory Detection of Fileless Malware 16 / 52 SBSEG’23

17. Introduction Proposed Solution Evaluation Conclusions
0x6.1 How do we detect malware?
Near-memory & In-Memory Detection of Fileless Malware 17 / 52 SBSEG’23

18. Introduction Proposed Solution Evaluation Conclusions
Publication
Figure: Source:
https://www.sciencedirect.com/science/article/pii/S0167404821003242
Near-memory & In-Memory Detection of Fileless Malware 18 / 52 SBSEG’23

19. Introduction Proposed Solution Evaluation Conclusions
AV Detection Mechanisms
Table: Deobfuscation Functions. Not all techniques are applied to entire payloads.
Technique XOR BASE64 RC4 Embedding/Carving
Mode Sig. RT OD Sig. RT OD Sig. RT OD Sign. RT OD
Avast 7 7 3 7 3 7 7 7 7
MalwareBytes 7 7 3 7 7 7 7 7 7
VIPRE 7 7 3 7 7 7 7 7 7
Kaspersky 7 7 3 3 3 7 7 7 7
TrendMicro 7 7 3 7 7 7 7 7 7
Near-memory & In-Memory Detection of Fileless Malware 19 / 52 SBSEG’23

20. Introduction Proposed Solution Evaluation Conclusions
Signatures as the Detection Mechanism
1 if( IsDebuggerPresent ()){
2 evade ()
Code 1: C code
1 mov eax , [fs:0x30]
2 mov eax , [eax+0x2]
3 jne 0 <evade >
Code 2: ASM code
1 64 8b 04 25 30 00 00
2 67 8b 40 02
3 75 e1
Code 3: Instructions Bytes
Near-memory & In-Memory Detection of Fileless Malware 20 / 52 SBSEG’23

21. Introduction Proposed Solution Evaluation Conclusions
0x6.1.1 Are signatures still widely-used?
Near-memory & In-Memory Detection of Fileless Malware 21 / 52 SBSEG’23

22. Introduction Proposed Solution Evaluation Conclusions
Signature Prevalence
0
5
10
15
20
25
30
35
40
45
AVG NOD32 Yandex GData DrWeb Emsisoft eScan AdAwe MAX BitDef Arcabit ZAlarm Kaspersky AhnLab Bkav Ikarus Microsoft Zillya ALYac NANOCybereason Avira Rising
Samples
(%)
AVs
AVs Detecting Specific Binary Sections
Figure: Signature Prevalence. Around a third of the AV’s detections are based on specific
section’s contents.
Near-memory & In-Memory Detection of Fileless Malware 22 / 52 SBSEG’23

23. Introduction Proposed Solution Evaluation Conclusions
0x6. Where does this overhead come from?
Near-memory & In-Memory Detection of Fileless Malware 23 / 52 SBSEG’23

24. Introduction Proposed Solution Evaluation Conclusions
Memory Dumping Techniques
10
100
1000
1 2 4 8
Time
(s)
RAM (GB)
Virtual−Proc Virtual−Full Physical
Figure: Memory dump time for distinct software-based techniques and memory sizes.
Near-memory & In-Memory Detection of Fileless Malware 24 / 52 SBSEG’23

25. Introduction Proposed Solution Evaluation Conclusions
0x7. Is there a way to eliminate this performance
cost?
Near-memory & In-Memory Detection of Fileless Malware 25 / 52 SBSEG’23

26. Introduction Proposed Solution Evaluation Conclusions
Publication
Figure: Source:
https://www.sciencedirect.com/science/article/abs/pii/S0957417422004882
Near-memory & In-Memory Detection of Fileless Malware 26 / 52 SBSEG’23

27. Introduction Proposed Solution Evaluation Conclusions
Understanding Malware Detection Tasks
Monitoring
You need to know: When to inspect.
Classifying
You need to know: What to inspect.
Near-memory & In-Memory Detection of Fileless Malware 27 / 52 SBSEG’23

28. Introduction Proposed Solution Evaluation Conclusions
Hardware-Enhanced AntiVirus Engine (HEAVEN)
2-level Architecture
Do not fully replace AVs, but add
efficient matching capabilities to them.
Near-memory & In-Memory Detection of Fileless Malware 28 / 52 SBSEG’23

29. Introduction Proposed Solution Evaluation Conclusions
0x8. Why not using existing hardware?
Near-memory & In-Memory Detection of Fileless Malware 29 / 52 SBSEG’23

30. Introduction Proposed Solution Evaluation Conclusions
Can’t We Rely on Page Faults?
Table: Blocking on Page Faults. The performance impact is greater as more complex is the
applied detection routine.
Benchmark Cycles PF 5K 10K 20K 30K
perf 187G 1,8M 4,74% 9,48% 18,96% 28,44%
mcf 69G 375K 2,72% 5,45% 10,89% 16,34%
milc 556G 1,2M 1,05% 2,10% 4,21% 6,31%
bzip 244G 170K 0,35% 0,69% 1,38% 2,08%
namd 491G 325K 0,33% 0,66% 1,32% 1,98%
Near-memory & In-Memory Detection of Fileless Malware 30 / 52 SBSEG’23

31. Introduction Proposed Solution Evaluation Conclusions
Agenda
1 Introduction
2 Proposed Solution
3 Evaluation
4 Conclusions
Near-memory & In-Memory Detection of Fileless Malware 31 / 52 SBSEG’23

32. Introduction Proposed Solution Evaluation Conclusions
Publication
Figure: Link: https://dl.acm.org/doi/10.1145/3422575.3422775
Near-memory & In-Memory Detection of Fileless Malware 32 / 52 SBSEG’23

33. Introduction Proposed Solution Evaluation Conclusions
0x9. How does the memory work?
What can we explore?
Near-memory & In-Memory Detection of Fileless Malware 33 / 52 SBSEG’23

34. Introduction Proposed Solution Evaluation Conclusions
Observing Memory Accesses Patterns
Figure: Write-to-Read window. Read requests originated from the MSHR might overlap
other memory-buffered read requests for any address, but must not overlap previous
memory-buffered write requests for the same address.
Near-memory & In-Memory Detection of Fileless Malware 34 / 52 SBSEG’23

35. Introduction Proposed Solution Evaluation Conclusions
0xA. How does the hardware detector look like?
Near-memory & In-Memory Detection of Fileless Malware 35 / 52 SBSEG’23

36. Introduction Proposed Solution Evaluation Conclusions
Malware Identification based on Near- and In-Memory Evaluation
(MINI-ME)
Figure: MINI-ME Architecture. MINI-ME is implemented within the memory controller.
Near-memory & In-Memory Detection of Fileless Malware 36 / 52 SBSEG’23

37. Introduction Proposed Solution Evaluation Conclusions
0xB. How does the software know about
hardware detections?
Near-memory & In-Memory Detection of Fileless Malware 37 / 52 SBSEG’23

38. Introduction Proposed Solution Evaluation Conclusions
Handling Notifications via Page Faults
1 void __do_page_fault (...) {
2 // Original Code
3 if (X86_PF_WRITE) ...
4 if (X86_PF_INSTR) ...
5 // Added Code
6 if (X86_MALICIOUS) ...
Code 4: Modified PF handler. Malicious bit is set
when suspicious pages are mapped.
Near-memory & In-Memory Detection of Fileless Malware 38 / 52 SBSEG’23

39. Introduction Proposed Solution Evaluation Conclusions
Agenda
1 Introduction
2 Proposed Solution
3 Evaluation
4 Conclusions
Near-memory & In-Memory Detection of Fileless Malware 39 / 52 SBSEG’23

40. Introduction Proposed Solution Evaluation Conclusions
0xC. How many CPU cycles can we delay?
Near-memory & In-Memory Detection of Fileless Malware 40 / 52 SBSEG’23

41. Introduction Proposed Solution Evaluation Conclusions
How Much Performance Overhead is Acceptable?
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
0 8 24 32 64 128
IPC
Overhead
(%)
Delay (Cycles)
IPC vs. Memory Delay
astar
calculix
dealII
gromacs
namd
Figure: MINI-ME database overhead. Delays of up 32 cycles impose less than 1% of IPC
overhead.
Near-memory & In-Memory Detection of Fileless Malware 41 / 52 SBSEG’23

42. Introduction Proposed Solution Evaluation Conclusions
0xD. What signature size should we use?
Near-memory & In-Memory Detection of Fileless Malware 42 / 52 SBSEG’23

43. Introduction Proposed Solution Evaluation Conclusions
Signature Size Definition
Table: Signature Generation. Signatures (%) detected as false positives for each signature
size and memory dump size.
Memory Size
1 GB 2 GB 4 GB 8 GB
Signature
Size
8 B 8.65% 9.92% 10.18% 11.45%
16 B 3.06% 3.32% 3.32% 3.32%
32 B 0.00% 0.00% 0.00% 0.00%
64 B 0.00% 0.00% 0.00% 0.00%
Near-memory & In-Memory Detection of Fileless Malware 43 / 52 SBSEG’23

44. Introduction Proposed Solution Evaluation Conclusions
0xE. Which storage type should we use?
Near-memory & In-Memory Detection of Fileless Malware 44 / 52 SBSEG’23

45. Introduction Proposed Solution Evaluation Conclusions
Matching Mechanism Definition
Table: Matching Techniques. FP rates for multiple signature sizes and techniques.
Signature size
8 B 16 B 32 B 64 B
Match.
Tech.
Dir. Mapped Table 8.33% 3.15% 0.00% 0.00%
Signature Tree 8.33% 3.15% 0.00% 0.00%
Bloom Filter 8.41% 3.47% 0.00% 0.00%
Near-memory & In-Memory Detection of Fileless Malware 45 / 52 SBSEG’23

46. Introduction Proposed Solution Evaluation Conclusions
0xF. What scan policy should we use?
Near-memory & In-Memory Detection of Fileless Malware 46 / 52 SBSEG’23

47. Introduction Proposed Solution Evaluation Conclusions
Matching Policies Definition
Table: Scan Policies. FP rate for multiple signature sizes and policies.
Signature size
8 B 16 B 32 B 64 B
Scan
Policy
Whole Memory 8.33% 3.15% 0.00% 0.00%
Mapped Pages 0.06% 0.01% 0.00% 0.00%
Whitelist 0.00% 0.00% 0.00% 0.00%
Code-Only 0.01% 0.00% 0.00% 0.00%
Near-memory & In-Memory Detection of Fileless Malware 47 / 52 SBSEG’23

48. Introduction Proposed Solution Evaluation Conclusions
0xF+1 (OOB). Is it better than a
software-based, on-access AV?
Near-memory & In-Memory Detection of Fileless Malware 48 / 52 SBSEG’23

49. Introduction Proposed Solution Evaluation Conclusions
MINI-ME in Practice
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
7.0%
8.0%
9.0%
10.0%
11.0%
12.0%
13.0%
perl namd bzip mcf milc
Execution
Time
Overhead
(%)
Monitoring Overhead
On−Access
MINI−ME
Figure: Monitoring Overhead. MINI-ME imposes a smaller overhead while still checking more
pages than an on-access solution.
Near-memory & In-Memory Detection of Fileless Malware 49 / 52 SBSEG’23

50. Introduction Proposed Solution Evaluation Conclusions
Agenda
1 Introduction
2 Proposed Solution
3 Evaluation
4 Conclusions
Near-memory & In-Memory Detection of Fileless Malware 50 / 52 SBSEG’23

51. Introduction Proposed Solution Evaluation Conclusions
Conclusions
Challenges & Lessons
Fileless malware is a growing hard-to-detect class of threats.
Traditional AntiViruses (AVs) impose significant performance overhead to perform
memory scans.
In-memory and Near-memory AVs helps reducing AV’s performance overheads.
The more complex the matching mechanism, the greater the performance
overhead.
MINI-ME as platform for future developments.
Near-memory & In-Memory Detection of Fileless Malware 51 / 52 SBSEG’23

52. Introduction Proposed Solution Evaluation Conclusions
Questions & Comments.
Contact
botacin@tamu.edu
@MarcusBotacin
Additional Material
https://github.com/marcusbotacin/In.Memory
https://marcusbotacin.github.io/
Looking Ahead
I’m looking for PhD students!
Near-memory & In-Memory Detection of Fileless Malware 52 / 52 SBSEG’23