Security crawlers and spiders crawl websites to test for vulnerabilities. A spider crawls a site by following links on sitemap files and pages, while a crawler also follows links but uses techniques like URL signatures to detect encoded or hidden pages. The document discusses techniques for testing SQL injection, XSS vulnerabilities, and other challenges faced by crawlers and spiders.

1. SECURITY
CRAWLER

2. …
crawl
Sitemap
bingbot https://www.keycdn.com/blog/web-
crawlers

3. Crawl
Sitema
p
spide
r

4. SPIDER VS CRAWLER
sitema
p
Spide
r
Sign

5. Spider
My
Project
Sql
inject
+
Sql
Injectio
n
XSS
LFI
Url
Signatur
e

6. My
project
Sign
Useag
e
abc.com/sport/%2d%3c%7d
%43
abc.com/category/encode
d
abc.com/news/how_to_use abc.com/category/title
abc.com/new_iran/fatf_may_201
8
abc.com/category/title
abc.com/how_to_use/145312 abc.com/title/number

7. Sign
Useag
e
sql
injectio
n
abc.com/sport/%2d%3c%7
d%43
abc.com/news/how_to_u
se
abc.com/sport/%2d%3c%7
d%43
Sign
abc.com/cat/encode
d
Sign
Sign
abc.com/cat/encode
d
Task
[URL
]
Task Task

8. useage
Sql
Injectio
n
XSS
Select * From Table Where ID = 1
http://abc.com/Table/?id=1
Select * From Table Where ID = 1 or 1=1
http://abc.com/Table/?id=1 or 1=1

9. sql
injectio
n
XSS
XSS

10. XSS
XSS
Questio
n
http://abc.com/Table/?id=<script>malicious code<script>

11. Xss
Question
s
challenge
s

12. questio
n
Challenge
s
challenge
s
spider
spider
spider
spider
spider
spider
spider
spider
Web
server

13. Challenge
s
Challenge
s
challenge
s
spider
spider
spider
Proxy
Proxy
Proxy
Web
server

14. Challenge
s
Challenge
s
challenge
s
spider
spider
spider
Web
server
Message
Message
Message
Message

15. Challenge
s
Challenge
s
Exec
abc.com/news/how_to_use abc.com/category/title
abc.com/new_iran/fatf_may_201
8
abc.com/category/title

16. Challenge
s
Exec
Questio
n

17. usage
Question
s
End

18. The
End