Build a Search Engine in a Weekend with Azure and F
•
2 j'aime•943 vues
5 of the most dangerous words you'll hear a developer say are "How hard could it be?". This talk tells the tale of what happens when you act on the question of "I'm going to write the next Google beater. How hard could it be?" This is the tale of how one person in a few hours is able to write something resembling a search engine thanks to the platform features of Azure and the productivity of F#. We'll see how we're able to use Azure search from F# to easily power our search internals, we'll use MBrace to rapidly find the most popular web pages on the internet and Azure functions to tie everything together to build up APIs and create on demand infrastructure. Add in a healthy mix of queues provided by Azure Service Bus and if you squint hard enough, you might just end up seeing something resembling a search engine. But seriously writing the next Google, just how hard could it be? A recording of this talk is available via SkillsMatter at https://skillsmatter.com/skillscasts/8901-f-sharpunctional-londoners-meetup
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (9)
Similaire à Build a Search Engine in a Weekend with Azure and F
Similaire à Build a Search Engine in a Weekend with Azure and F (20)
Dernier
Dernier (20)
Build a Search Engine in a Weekend with Azure and F
- 1. WRITING A SEARCH ENGINE. HOW HARD COULD IT BE? ANTHONY BROWN @BRUINBROWN93 ANTHONY@COMPOSITIONAL-IT.COM
- 2. ABOUT ABOUT ME ▸ Consultant at Compositional IT ▸ F# dev for ~3 years now ▸ Interested in Big Data, IoT, Cloud and Distributed Systems
- 3. COMPOSITIONAL IT FUNCTIONAL FIRST. CLOUD READY. @COMPOSITIONALIT
- 4. HOW HARD COULD IT BE? Every software developer ever INTRODUCTION
- 5. IT’S ONLY AN OPERATING SYSTEM, ALL IT DOES IS RUNS PROGRAMS! Everybody when Windows blue screens INTRODUCTION
- 6. IT’S ONLY A MULTIPLAYER ONLINE VIDEO GAME! Anybody playing a game when lag spikes hit TEXT
- 7. IT’S ONLY 2 LINES OF JAVASCRIPT Backend developer needing to make a small API change INTRODUCTION
- 8. DUDE. HOLD MY BEER. Drunk people 10 seconds before making a terrible mistake
- 10. WHAT NEXT? HIT UP GOOGLE.
- 11. WHAT TO DO IN LONDON THIS WEEKEND?
- 14. WRITING A SEARCH ENGINE. HOW HARD COULD IT BE?
- 15. WRITING A SEARCH ENGINE WITH AZURE AND F# IN A WEEKEND.
- 16. BUT FIRST.
- 17. THIS WAS A WEEKEND PROJECT.
- 18. YOU SHOULD EXPECT: - HACKY CODE.
- 19. YOU SHOULD EXPECT: - DEMOS TO FAIL.
- 20. YOU SHOULD NOT EXPECT: - A DEEP DIVE INTO SEARCH ENGINE TECH.
- 21. SEARCH ENGINE BACKGROUND CONSTRAINTS ▸ Not a priority ▸ Can’t cost more than £85 per month ▸ No operations investment ▸ Limit to the weekend
- 22. BACKGROUND EVERYTHING I KNOW ABOUT HOW SEARCH ENGINES WORK ▸ ▸ ▸ ▸ ▸ ▸
- 23. THE ANATOMY OF A LARGE- SCALE HYPER TEXTUAL WEB SEARCH ENGINE SERGEY BRIN LARRY PAGE
- 25. IT’S 2016. THE WEB’S CHANGED. A LOT.
- 30. WHAT’S NEW? + PLATFORM AS A SERVICE
- 32. WHAT’S NEW? - PERSONAL HOSTING
- 33. SEARCH ENGINE BACKGROUND WHAT’S IMPORTANT? ▸ Search ▸ Scraping ▸ Page rank
- 34. SEARCH IMPLEMENTATION HOW TO FIND A NEEDLE IN A HAYSTACK ▸ Take all of your documents ▸ Record all of the words which occur within a file ▸ Invert that index ▸ List of all words and the documents they appear in ▸ For all words in the search query, find the files which appear in every inverted index
- 35. SOUNDS EASY RIGHT? I DON’T CARE ABOUT IT.
- 36. AZURE SEARCH MANAGED SEARCH AS A SERVICE
- 37. AZURE SEARCH WHAT DOES AZURE SEARCH GIVE US? ▸ Hosted Search as a Service ▸ HTTP API for indexing and retrieving documents ▸ Ability to scale out (more replicas, more indexes) ▸ Free basic tier
- 38. AZURE SEARCH IN THE AZURE PORTAL.
- 39. BOOSTING DEMO.
- 40. WE HAVE SEARCH. WHAT NEXT?
- 41. INDEXING DATA WHAT IS A CRAWLER ▸ Autonomously find every web page on the internet ▸ Pull the content from that web page and index it ▸ Read the links on that page and index those links ▸ Recursively process until every page on the internet has been reached
- 42. THE PROBLEM? THE INTERNET’S PRETTY BIG.
- 43. AZURE SERVICE BUS DISTRIBUTED MESSAGE QUEUES
- 44. INDEXING DATA WHAT DOES AZURE SERVICE BUS GIVE US? ▸ Scalable durable queues and topics with guaranteed availability ▸ .Net APIs to communicate with the service bus ▸ Free basic tier
- 45. WORKING WITH A SERVICE BUS QUEUE.
- 46. WE NEED TO BE GOOD CITIZENS. WE DON’T WANT TO DDOS A SINGLE WEBSITE DURING CRAWLING.
- 47. SERVICE BUS PROVIDES SUPPORT FOR MESSAGE DE-DUPLICATION BASED ON CONTENT.
- 48. WE DON’T WANT TO SCRAPE THROUGH EVERY WEB PAGE IN THE WORLD.
- 49. WE DON’T WANT TO INDEX: - GOOGLE SEARCH QUERIES
- 50. WE DON’T WANT TO INDEX: - PROTECTED CONTENT
- 51. WE DON’T WANT TO INDEX: - IRRELEVANT CONTENT
- 52. DEALING WITH THE ROBOTS.TXT FILE WRITING BASIC PARSERS IN F#
- 53. BEING A WELL BEHAVED SCRAPER WHAT IS ROBOTS.TXT? ▸ Text file standard for telling web scrapers what they should scrape ▸ Opt-in - crawlers can ignore the robots.txt file ▸ Simple file stored at the root of the web server
- 56. HTML AND INFORMATION RETRIEVAL QUERYING HTML DOCUMENTS WITH HTML AGILITY PACK
- 57. WE HAVE A HTML FILE. WE NEED THE CONTENT OUT OF IT.
- 58. INFORMATION RETRIEVAL FROM HTML DOCUMENTS WORKING WITH THE HTML AGILITY PACK ▸ Provides a simple query layer over HTML documents ▸ Works with well formatted and poorly formatted HTML ▸ Provides XPath support over the document ▸ Allows for querying for individual properties and elements
- 59. EXTRACTING LINKS FROM A HTML DOCUMENT
- 60. EXTRACTING ALL OF THE CONTENT FROM AN HTML DOCUMENT
- 61. WE NOW HAVE A WEB SCRAPER. WE NEED TO RUN THE WEB SCRAPER.
- 62. AZURE WEBJOBS SIMPLE HOSTING OF LONG RUNNING PROCESSES
- 63. AZURE WEB JOBS WHAT ARE WEB JOBS? ▸ A means of hosting basic executables in the cloud ▸ Provides simplified deployment and monitoring ▸ Pricing per minute of usage
- 64. WE NOW HAVE A SEARCH ENGINE. KIND OF.
- 65. SEARCH IS A RECOMMENDATION PROBLEM.
- 66. HOW DO WE RECOMMEND CONTENT TO USERS?
- 67. PAGE RANK FINDING THE MOST INFLUENTIAL SITES ON THE INTERNET
- 68. PAGE RANK WHAT IS PAGE RANK? ▸ Stanford’s patented algorithm ▸ Helps you find the most influential websites on the internet ▸ Websites with lots of links to them are more influential
- 69. THE PROBLEM? THERE’S LOTS OF WEBSITES ON THE INTERNET.
- 70. THERE’S EVEN MORE LINKS BETWEEN WEBSITES.
- 71. WE HAVE A HUGE LINK GRAPH. WE NEED TO PROCESS THAT GRAPH.
- 72. BIG DATA PROCESSING WITH MBRACE AND CLOUDFLOWS.
- 73. WE HAVE A QUERY WHICH NEEDS TO RUN DAILY. WE NEED TO ORCHESTRATE IT.
- 74. AZURE FUNCTIONS + AZURE RESOURCE MANAGER USING AZURE FUNCTIONS FOR DEVOPS
- 75. DEVOPS WHAT IS AZURE RESOURCE MANAGER? ▸ Declarative way of describing Azure infrastructure ▸ REST APIs to deploy infrastructure template files ▸ APIs to see current deployment status
- 76. DEVOPS WHAT IS AZURE FUNCTIONS? ▸ Lightweight scripting of Azure web jobs ▸ Allows for running scripts in response to certain events ▸ Billing based on number of function invocations
- 77. DEVOPS USING AZURE FUNCTIONS FOR DEVOPS ▸ Set up a timer triggered Azure Function ▸ Deploy an Mbrace cluster through Azure Resource Manager ▸ Send an event when the job completes ▸ Second Azure Function for deleting the MBrace cluster
- 78. AZURE FUNCTIONS AND AZURE RESOURCE MANAGER.
- 79. WE NOW HAVE EVERYTHING IN PLACE FOR A SEARCH ENGINE. NOBODY CAN ACCESS IT THOUGH.
- 80. AZURE FUNCTIONS SERVERLESS WEB APIS WITH AZURE FUNCTIONS
- 81. AZURE FUNCTIONS CAN OPERATE ON HTTP REQUESTS.
- 82. NO LONG TERM HOSTING COSTS.
- 83. AZURE FUNCTIONS HTTP API DEMO.
- 85. HTTP API AZURE SEARCH LINK DATABASE PAGERANK CLUSTER ORCHESTRATOR AZURE SERVICEBUS INDEXER PAGERANK IMPORTER PAGERANK SCORE STORE
- 86. PLENTY OF ROOM FOR IMPROVEMENTS.
- 91. SUPPORT FOR OTHER DOCUMENT TYPES.
- 93. WHAT’S NEXT FOR IT? NOTHING.
- 94. PRODUCTISING A GOOGLE COMPETITOR IS BASICALLY IMPOSSIBLE.
- 95. IN SUMMARY WRAPPING UP & KEY TAKEAWAYS
- 96. AZURE + F# = <3
- 97. AZURE MAKES HARD INFRASTRUCTURE PROBLEMS SIMPLE.
- 98. F# MAKES HARD SOFTWARE PROBLEMS SIMPLE.
- 99. TOGETHER THEY MAKE HARD PROBLEMS SIMPLE.
- 100. IT’S NOT GOOGLE. BUT IT TOOK 1 DEV 2 DAYS.
- 101. CLOUD IS THE EPITOME OF BUSINESS AGILITY
- 103. COMPOSITIONAL IT ANTHONY@COMPOSITIONAL-IT.COM FUNCTIONAL FIRST. CLOUD READY.
- 104. Q&A.