Traditional file transfer is still an important backbone of operations for many organizations. Because of this, quick file processing, for instance by a high available setup of an enterprise service bus (ESB) cluster, is a goal for many. WSO2 offers the core functionality for this and Yenlo has added 24/7 support for high availability, guaranteed delivery , content and metadata based routing in a multitenant and a scalable environment. In this session Yenlo presents the advantages of managed file transfer with WSO2 Enterprise Service Bus (WSO2 ESB) and shares experience gained in real world setups.

1. Managed File Transfer
anno 2016, old fashioned stuff
Jochen Trauncker & Ruben van der Zwan
Yenlo

2. Managed File Transfer
§ Yenlo Labs ini<a<ve
§ Yenlo Labs is R&D of Yenlo
§ Yenlo is an interna<onal opera<ng
integra<on specialist and Premier Partner of
WSO2
§ Goal: simplify integra<on of files

3. It is 2016. Do we s<ll need to talk about file transfer?
• There are 45508 companies listed in the stock exchanges
§ There are in total 115M companies registered world wide
• Most of them s<ll use files

4. Companies love files
§ Big files (Gigabytes)
§ Small files (just a few Kilobytes)
§ Only a few files every once in a while
§ Many files in one folder
§ Many folders with many files
§ And so on

5. Files can be mean fellows
§ Encoding (>100K Q&A on Stackoverflow)
§ Java and File (>200K Q&A on Stackoverflow)
§ Business seman<cs in file names
§ File system specific idiosyncrasies
§ Permissions
§ Locking
§ Concurrency

6. Example: when can a file get „processed“ ?
§ Visibility in the filesystem is a bad idea
§ Atomic opera<on
§ “mv" from some “hidden” loca<on
§ File name conven<on
§ “abc.xml.s<llwri<ng”
§ mv ”abc.xml.s<llwri<ng à abc.xml”
§ Lock-File
§ ”abc.xml” & “abc.lck”
§ File size delta
§ Well defined point in <me

7. Example: when can a file get „processed“ ?
§ Ul<mate file provider & consumers
§ Mostly legacy
§ No way to change them
§ MFT has to take care of
§ File locking
§ File size monitoring
§ Atomic file movement
§ And so on

8. And finally the transport of files should be ...
§ 100% guaranteed, performant and reliable
§ Encrypted, archived, monitored
§ Ordered, reordered, controlled and auditable

9. Managed File Transfer is taking care of ...
§ Transport
§ Security
§ Compliancy
§ Control
§ Workflow integra<on

10. Transport: local
§ copy / move between directories
§ one host with several mount points

11. Transport: host-to-host
§ transfer content via
§ HTTP(S)
§ (S)FTP
§ ….
§ manage source / target loca<on
§ via transferred meta-data
§ up-front configura<on (conven<on, content, …)
§ direct coupling of source and target

12. Transport: host-queue-host
§ transfer content via queue as
intermediary
§ decouples source from target

13. Transport: host-intermediary-host
§ intermediary provides
§ intermediary storage
§ management of the transfer
§ helpful for
§ (re)ordering of delivery
§ <med delivery
§ merging of files

14. Security
§ Deliver files to en<tled audience
§ Malicious content
§ DoS amacks
§ Too many files
§ Too big files
§ Blacklis<ng
§ Whitelis<ng

15. Compliancy
§ Guaranteed delivery
§ Timely delivery
§ Audi<ng and Logging
§ Encryp<on
§ Checksumming

16. Control
§ Processing order
§ Delivery order
§ Timing
§ Rou<ng
§ Parallelizing
§ Synchronizing

17. Workflow integra<on
§ Long-running processes
§ Triggering several subsystems asynchronous
§ Human task integra<on

18. So that is what we did….
§ We started building an MFT
solu<on on top of WSO2
§ Based on:
§ WSO2 ESB
§ WSO2 Message Broker
§ WSO2 Data Analy<cs Server
§ WSO2 Business Process Server

19. Key features
§ Rou<ng based on name or content
§ Push and Pull methods for (S)FTP, SMB, and more…
§ Guaranteed delivery (+Delivery Retry mechanism)
§ Automa<c archiving
§ Detec<on of double file input
§ Single source, mul<ple Targets
§ Determinate sending order of files to Target(s)
§ Monitoring and Aler<ng op<ons

20. Where do we stand?
§ The run<me is (almost) ready
§ The user-interface (web-app) is work in progress….
§ Extensive tes<ng on various file systems
§ Aim to launch full solu<on end of 2016

21. Some remaining challenges
§ Chunking of (large) files
§ Streaming of (large) files through several intermediaries
§ Individual file system implementa<ons and their proper<es
§ Deduc<on of file encoding

22. Final note
“Never underes=mate the
bandwidth of a sta=on wagon full of
tapes hurtling down the highway.”
Andrew S. Tanenbaum

23. Thank You!
#WSO2ConEU
Share your feedback for this session
wso2con.com/app