Ivan Kosianenko discusses how AppsFlyer has evolved its data pipeline from using MemSQL and Druid to handle 12 billion events daily to now using ClickHouse and Druid to handle 38 billion events daily. He then explains how building a custom data streaming solution with Clojure can become complex due to issues like distributed orchestration, monitoring, and delivery guarantees. As an alternative, Ivan recommends using Spark Structured Streaming for its built-in functionality addressing these issues like write-ahead logging and exactly-once processing. He demonstrates how AppsFlyer leverages Spark Structured Streaming to ingest streaming data from Kafka into ClickHouse while monitoring streaming queries and end-to-end latency.