How to manage large amounts of data with Iteratee - ScalaDays Berlin 2014
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

How to manage large amounts of data with Iteratee - ScalaDays Berlin 2014

le

  • 1,366 vues

How to manage a large file with HTTP? Is it possible to manage data stream when you are in a HTTP POST request? How to be relax when managing data stream (ie : not use while(true) hack or something ...

How to manage a large file with HTTP? Is it possible to manage data stream when you are in a HTTP POST request? How to be relax when managing data stream (ie : not use while(true) hack or something I/O block)?

This talk is about Play! iteratee and how to use it on real project : what is an iteratee? why it's useful? how to use it? how to manage it? is it clean? is your code readable or not?

http://scaladays.org/#schedule/How-to-manage-large-amounts-of-data-with-Iteratee

Statistiques

Vues

Total des vues
1,366
Vues sur SlideShare
1,312
Vues externes
54

Actions

J'aime
7
Téléchargements
26
Commentaires
0

6 Ajouts 54

https://www.blogger.com 19
http://chemistryband1.blogspot.com 19
https://chemistryband1.blogspot.com 7
http://cb591.blogspot.com 6
https://www.linkedin.com 2
https://twitter.com 1

Accessibilité

Catégories

Détails de l'import

Uploaded via as Microsoft PowerPoint

Droits d'utilisation

© Tous droits réservés

Report content

Signalé comme inapproprié Signaler comme inapproprié
Signaler comme inapproprié

Indiquez la raison pour laquelle vous avez signalé cette présentation comme n'étant pas appropriée.

Annuler
  • Full Name Full Name Comment goes here.
    Êtes-vous sûr de vouloir
    Votre message apparaîtra ici
    Processing...
Poster un commentaire
Modifier votre commentaire

How to manage large amounts of data with Iteratee - ScalaDays Berlin 2014 Presentation Transcript

  • 1. HOW TO MANAGE LARGE AMOUNTS OF DATA WITH ITERATEE BY @WAXZCE – QUENTIN ADAM SCALA DAYS 2014 BERLIN
  • 2. MY DAY TO DAY WORK : CLEVER CLOUD, MAKE YOUR APP RUN ALL THE TIME
  • 3. And learn a lot of things about your code, apps, and good/bad design… KEEP YOUR APPS ONLINE. MADE WITH NODE.JS, SCALA, JAVA, RUBY, PHP, PYTHON, GO…
  • 4. AND LEARN A LOT OF THINGS ABOUT YOUR CODE, APPS, AND GOOD/BAD DESIGN…
  • 5. WHY STREAMS ARE SO TRENDY THESE DAYS?
  • 6. 2 MAIN REASONS
  • 7. DATA ARE COMING BIGGER AND BIGGER
  • 8. WE NEED TO ACT WHILE DATA TRANSFERRING IS OCCURRING
  • 9. &
  • 10. WE ARE NOW USING LOTS OF PERMANENT CONNECTION
  • 11. EXAMPLE
  • 12. WHAT IS INSIDE AN HTTP REQUEST ? Verb • The action Resource • The object of the action Headers • The context of the action Body • Optional • The datas
  • 13. IN MANY CASE THE REQUEST IS MANIPULATE ALL FROM MEMORY
  • 14. UPLOADING A 20 GB FILE ON A POST REQUEST AND STORE IT EXAMPLE
  • 15. IT’S A BAD IDEA TO PUT THE BODY PART IN MEMORY
  • 16. CREATE A TEMP FILE TO STORE THE DATA
  • 17. Client upload file Create a temp file Send a file in a backend Then answer to the client The file is transferring two times
  • 18. NOT SO GOOD
  • 19. Client upload file Directly stream it to the backend Then answer to the client
  • 20. LET’S ACT ON STREAM!
  • 21. CLASSIC JAVA STREAM MANAGEMENT
  • 22. CLASSIC JAVA STREAM MANAGEMENT • Buffers • Buffer management • Buffer exception • Buffer overflow
  • 23. CLASSIC JAVA STREAM MANAGEMENT • Low performances if not buffered • Not modular • Thread blocking • Code is ugly • No back pressure • Error handling is bad • i/o management and business code are mixed
  • 24. I CAN’T SLEEP IN THE NIGHT
  • 25. WE HAVE TO FIND A BETTER WAY
  • 26. THAT’S WHY WE NEED ITERATEE
  • 27. DATA STRUCTURES TO EXPRESS DATA STREAM MANAGEMENT
  • 28. Like a recipe Consume the data ITERATEE : HOW TO MANAGE A STREAM
  • 29. Produce the data ENUMERATOR : DATA STREAM
  • 30. Set of tools to do cool things with Iteratee and Enumerator ENUMERATEE
  • 31. SIMPLE EXAMPLE TO START
  • 32. AN ENUMERATOR
  • 33. AN ITERATEE
  • 34. INPUTS • EOF • Empty • El
  • 35. ITERATEE STATES • Done • Error • Cont
  • 36. AN ITERATEE
  • 37. AN ITERATEE TO MANAGE THE STREAM
  • 38. AN ITERATEE
  • 39. ITERATEE COMPOSITION
  • 40. PURE FUNCTIONAL
  • 41. TYPE SAFE
  • 42. COMPOSITION
  • 43. SO WE WILL JUST MANAGE THE BODY STREAM JUST DO NOT REWRITE HTTP PARSER
  • 44. CREATE TEMP FILE Built in on play with
  • 45. CREATE TEMP FILE Built in on play with
  • 46. BODY PARSERS REQUEST HEADERS -> ITERATEE[ARRAY[BYTE], EITHER[SIMPLERESULT, ?]]
  • 47. GET FILE AND CALCULATE HASH FROM CHUNK
  • 48. DEMO
  • 49. I WANT TO USE FORMS
  • 50. SO PARTHANDLER ALLOW YOU TO COMPOSE A BODYPARSER ITERATEE <3 COMPOSITION
  • 51. NOW WE CAN ACCESS TO STREAMS 
  • 52. DATA STORAGE
  • 53. I HATE FILE SYSTEM
  • 54. I HATE FILE SYSTEM
  • 55. DO NOT USE THE FILE SYSTEM AS A DATASTORE File system are POSIX compliant • POSIX is ACID • POSIX is powerful but is a bottleneck • File System is the nightmare of ops • File System creates coupling (host provider/OS/language) • SPOF-free multi tenant File System is a unicorn
  • 56. • Key = hash of the chunk STORE FILE IN CHUNK INSIDE K/V STORE
  • 57. STORE AS META DATA THE LIST OF CHUNKS
  • 58. PROCESS File is post by browser Stream is chunk nomalized Each chunk is async store in kv List of chunk hash is store with file id No overhead for data store 
  • 59. NEW DEMO • Play 2 + iteratee • Datastore is riak 
  • 60. ARE YOU CONVINCE TO USE STREAM ?
  • 61. DATA STREAMING PROGRAMMING IS NOW TRENDING
  • 62. THX TO @CLEMENTD CLEMENT DELAFARGUE
  • 63. I’m @waxzce on twitter I’m the CEO of A PaaS provider, give it a try ;-) THX FOR LISTENING & QUESTIONS TIME Coupon for Clever Cloud trial : berlin_scala_days