SlideShare une entreprise Scribd logo
1  sur  94
Télécharger pour lire hors ligne
@crichardson
Map(), flatMap() and reduce() are your
new best friends:
Simpler collections, concurrency, and big
data
Chris Richardson
Author of POJOs in Action
Founder of the original CloudFoundry.com
@crichardson
chris@chrisrichardson.net
http://plainoldobjects.com
@crichardson
Presentation goal
How functional programming simplifies
your code
Show that
map(), flatMap() and reduce()
are remarkably versatile functions
@crichardson
About Chris
@crichardson
About Chris
Founder of a buzzword compliant (stealthy, social, mobile,
big data, machine learning, ...) startup
Consultant helping organizations improve how they
architect and deploy applications using cloud, micro
services, polyglot applications, NoSQL, ...
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
What’s functional
programming?
@crichardson
It’s a programming paradigm
@crichardson
It’s a kind of programming
language
@crichardson
Functions as the building
blocks of the application
@crichardson
Functions as first class
citizens
Assign functions to variables
Store functions in fields
Use and write higher-order functions:
Pass functions as arguments
Return functions as values
@crichardson
Avoids mutable state
Use:
Immutable data structures
Single assignment variables
Some functional languages such as Haskell don’t side-effects
There are benefits to immutability
Easier concurrency
More reliable code
But be pragmatic
@crichardson
Why functional programming?
"the highest goal of
programming-language
design to enable good
ideas to be elegantly
expressed"
http://en.wikipedia.org/wiki/Tony_Hoare
@crichardson
Why functional programming?
More expressive
More concise
More intuitive - solution matches problem definition
Elimination of error-prone mutable state
Easy parallelization
@crichardson
An ancient idea that has
recently become popular
@crichardson
Mathematical foundation:
λ-calculus
Introduced by
Alonzo Church in the 1930s
@crichardson
Lisp = an early functional
language invented in 1958
http://en.wikipedia.org/wiki/
Lisp_(programming_language)
1940
1950
1960
1970
1980
1990
2000
2010
garbage collection
dynamic typing
self-hosting compiler
tree data structures
(defun factorial (n)
(if (<= n 1)
1
(* n (factorial (- n 1)))))
@crichardson
My final year project in 1985:
Implementing SASL
sieve (p:xs) =
p : sieve [x | x <- xs, rem x p > 0];
primes = sieve [2..]
A list of integers starting with 2
Filter out multiples of p
Mostly an Ivory Tower
technology
Lisp was used for AI
FP languages: Miranda,
ML, Haskell, ...
“Side-effects
kills kittens and
puppies”
@crichardson
http://steve-yegge.blogspot.com/2010/12/haskell-researchers-announce-discovery.html
!*
!*
!*
@crichardson
But today FP is mainstream
Clojure - a dialect of Lisp
A hybrid OO/functional language
A hybrid OO/FP language for .NET
Java 8 has lambda expressions
@crichardson
Java 8 lambda expressions
are functions
x -> x * x
x -> {
for (int i = 2; i < Math.sqrt(x); i = i + 1) {
if (x % i == 0)
return false;
}
return true;
};
(x, y) -> x * x + y * y
@crichardson
Java 8 lambdas are a
shorthand* for an anonymous
inner class
* not exactly. See http://programmers.stackexchange.com/questions/
177879/type-inference-in-java-8
@crichardson
Java 8 functional interfaces
Interface with a single abstract method
e.g. Runnable, Callable, Spring’s TransactionCallback
A lambda expression is an instance of a functional
interface.
You can use a lambda wherever a function interface
“value” is expected
The type of the lambda expression is determined from it’s
context
@crichardson
Example Functional Interface
Function<Integer, Integer> square = x -> x * x;
BiFunction<Integer, Integer, Integer>
sumSquares = (x, y) -> x * x + y * y;
Predicate<Integer> makeIsDivisibleBy(int y) {
return x -> x % y == 0;
}
Predicate<Integer> isEven = makeIsDivisibleBy(2);
Assert.assertTrue(isEven.test(8));
Assert.assertFalse(isEven.test(11));
@crichardson
Example Functional Interface
ExecutorService executor = ...;
final int x = 999
Future<Boolean> outcome = executor.submit(() -> {
for (int i = 2; i < Math.sqrt(x); i = i + 1) {
if (x % i == 0)
return false;
}
return true;
}
This lambda is
a Callable
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
Lot’s of application code
=
collection processing:
Mapping, filtering, and reducing
@crichardson
Social network example
public class Person {
enum Gender { MALE, FEMALE }
private Name name;
private LocalDate birthday;
private Gender gender;
private Hometown hometown;
private Set<Friend> friends = new HashSet<Friend>();
....
public class Friend {
private Person friend;
private LocalDate becameFriends;
...
}
public class SocialNetwork {
private Set<Person> people;
...
@crichardson
Mapping, filtering, and
reducing
public class Person {
public Set<Hometown> hometownsOfFriends() {
Set<Hometown> result = new HashSet<>();
for (Friend friend : friends) {
result.add(friend.getPerson().getHometown());
}
return result;
}
@crichardson
Mapping, filtering, and
reducing
public class Person {
public Set<Person> friendOfFriends() {
Set<Person> result = new HashSet();
for (Friend friend : friends)
for (Friend friendOfFriend : friend.getPerson().friends)
if (friendOfFriend.getPerson() != this)
result.add(friendOfFriend.getPerson());
return result;
}
@crichardson
Mapping, filtering, and
reducing
public class SocialNetwork {
private Set<Person> people;
...
public Set<Person> lonelyPeople() {
Set<Person> result = new HashSet<Person>();
for (Person p : people) {
if (p.getFriends().isEmpty())
result.add(p);
}
return result;
}
@crichardson
Mapping, filtering, and
reducing
public class SocialNetwork {
private Set<Person> people;
...
public int averageNumberOfFriends() {
int sum = 0;
for (Person p : people) {
sum += p.getFriends().size();
}
return sum / people.size();
}
@crichardson
Problems with this style of
programming
Low level
Imperative (how to do it) NOT declarative (what to do)
Verbose
Mutable variables are potentially error prone
Difficult to parallelize
@crichardson
Java 8 streams to the rescue
A sequence of elements
“Wrapper” around a collection
Streams can also be infinite
Provides a functional/lambda-based API for transforming,
filtering and aggregating elements
Much simpler, cleaner code
@crichardson
Using Java 8 streams -
mapping
class Person ..
private Set<Friend> friends = ...;
public Set<Hometown> hometownsOfFriends() {
return friends.stream()
.map(f -> f.getPerson().getHometown())
.collect(Collectors.toSet());
}
@crichardson
The map() function
s1 a b c d e ...
s2 f(a) f(b) f(c) f(d) f(e) ...
s2 = s1.map(f)
@crichardson
public class SocialNetwork {
private Set<Person> people;
...
public Set<Person> peopleWithNoFriends() {
Set<Person> result = new HashSet<Person>();
for (Person p : people) {
if (p.getFriends().isEmpty())
result.add(p);
}
return result;
}
Using Java 8 streams -
filtering
public class SocialNetwork {
private Set<Person> people;
...
public Set<Person> lonelyPeople() {
return people.stream()
.filter(p -> p.getFriends().isEmpty())
.collect(Collectors.toSet());
}
@crichardson
Using Java 8 streams - friend
of friends V1
class Person ..
public Set<Person> friendOfFriends() {
Set<Set<Friend>> fof = friends.stream()
.map(friend -> friend.getPerson().friends)
.collect(Collectors.toSet());
...
}
Using map()
=> Set of Sets :-(
Somehow we need to flatten
@crichardson
Using Java 8 streams -
mapping
class Person ..
public Set<Person> friendOfFriends() {
return friends.stream()
.flatMap(friend -> friend.getPerson().friends.stream())
.map(Friend::getPerson)
.filter(f -> f != this)
.collect(Collectors.toSet());
}
maps and flattens
@crichardson
The flatMap() function
s1 a b ...
s2 f(a)0 f(a)1 f(b)0 f(b)1 f(b)2 ...
s2 = s1.flatMap(f)
@crichardson
Using Java 8 streams -
reducing
public class SocialNetwork {
private Set<Person> people;
...
public long averageNumberOfFriends() {
return people.stream()
.map ( p -> p.getFriends().size() )
.reduce(0, (x, y) -> x + y)
/ people.size();
} int x = 0;
for (int y : inputStream)
x = x + y
return x;
@crichardson
The reduce() function
s1 a b c d e ...
x = s1.reduce(initial, f)
f(f(f(f(f(f(initial, a), b), c), d), e), ...)
@crichardson
Newton's method for finding square
roots
public class SquareRootCalculator {
public double squareRoot(double input, double precision) {
return
Stream.iterate(
new Result(1.0),
current -> refine(current, input, precision))
.filter(r -> r.done)
.findFirst().get().value;
}
private static Result refine(Result current,
double input, double precision) {
double value = current.value;
double newCurrent =
value - (value * value - input) / (2 * value);
boolean done = Math.abs(value - newCurrent) < precision;
return new Result(newCurrent, done);
}
class Result { boolean done; double value; }
Creates an infinite stream:
seed, f(seed), f(f(seed)), .....
Don’t panic!
Streams are lazy
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
Tony’s $1B mistake
“I call it my billion-dollar mistake.
It was the invention of the null
reference in 1965....But I couldn't
resist the temptation to put in a
null reference, simply because it
was so easy to implement...”
http://qconlondon.com/london-2009/presentation/
Null+References:+The+Billion+Dollar+Mistake
@crichardson
Coding with null pointers
class Person
public Friend longestFriendship() {
Friend result = null;
for (Friend friend : friends) {
if (result == null ||
friend.getBecameFriends()
.isBefore(result.getBecameFriends()))
result = friend;
}
return result;
}
Friend oldestFriend = person.longestFriendship();
if (oldestFriend != null) {
...
} else {
...
}
Null check is essential yet
easily forgotten
@crichardson
Java 8 Optional<T>
A wrapper for nullable references
It has two states:
empty throws an exception if you try to get the reference
non-empty contain a non-null reference
Provides methods for:
testing whether it has a value
getting the value
...
Return reference wrapped in an instance of this type instead of null
@crichardson
Coding with optionals
class Person
public Optional<Friend> longestFriendship() {
Friend result = null;
for (Friend friend : friends) {
if (result == null ||
friend.getBecameFriends().isBefore(result.getBecameFriends()))
result = friend;
}
return Optional.ofNullable(result);
}
Optional<Friend> oldestFriend = person.longestFriendship();
// Might throw java.util.NoSuchElementException: No value present
// Person dangerous = popularPerson.get();
if (oldestFriend.isPresent) {
...oldestFriend.get()
} else {
...
}
@crichardson
Using Optionals - better
Optional<Friend> oldestFriendship = ...;
Friend whoToCall1 = oldestFriendship.orElse(mother);
Avoid calling isPresent() and get()
Friend whoToCall3 =
oldestFriendship.orElseThrow(
() -> new LonelyPersonException());
Friend whoToCall2 =
oldestFriendship.orElseGet(() -> lazilyFindSomeoneElse());
@crichardson
Using Optional.map()
public class Person {
public Optional<Friend> longestFriendship() {
return ...;
}
public Optional<Long> ageDifferenceWithOldestFriend() {
Optional<Friend> oldestFriend = longestFriendship();
return oldestFriend.map ( of ->
Math.abs(of.getPerson().getAge() - getAge())) );
}
Eliminates messy conditional logic
@crichardson
Using flatMap()
class Person
public Optional<Friend> longestFriendship() {...}
public Optional<Friend> longestFriendshipOfLongestFriend() {
return
longestFriendship()
.flatMap(friend ->
friend.getPerson().longestFriendship());
}
not always a symmetric
relationship. :-)
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
Let’s imagine you are performing
a CPU intensive operation
class Person ..
public Set<Hometown> hometownsOfFriends() {
return friends.stream()
.map(f -> cpuIntensiveOperation())
.collect(Collectors.toSet());
}
@crichardson
class Person ..
public Set<Hometown> hometownsOfFriends() {
return friends.parallelStream()
.map(f -> cpuIntensiveOperation())
.collect(Collectors.toSet());
}
Parallel streams = simple
concurrency Potentially uses N cores
Nx speed up
@crichardson
Let’s imagine that you are
writing code to display the
products in a user’s wish list
@crichardson
The need for concurrency
Step #1
Web service request to get the user profile including wish
list (list of product Ids)
Step #2
For each productId: web service request to get product info
Sequentially terrible response time
Need fetch productInfo concurrently
@crichardson
Futures are a great
concurrency abstraction
http://en.wikipedia.org/wiki/Futures_and_promises
@crichardson
Worker thread or
event-driven
Main thread
How futures work
Outcome
Future
Client
get
Asynchronous
operation
set
initiates
@crichardson
Benefits
Simple way for two concurrent activities to communicate safely
Abstraction:
Client does not know how the asynchronous operation is
implemented
Easy to implement scatter/gather:
Scatter: Client can invoke multiple asynchronous operations
and gets a Future for each one.
Gather: Get values from the futures
@crichardson
Example wish list service
public interface UserService {
Future<UserProfile> getUserProfile(long userId);
}
public class UserServiceProxy implements UserService {
private ExecutorService executorService;
@Override
public Future<UserProfile> getUserProfile(long userId) {
return executorService.submit(() ->
restfulGet("http://uservice/user/" + userId,
UserProfile.class));
}
...
}
public interface ProductInfoService {
Future<ProductInfo> getProductInfo(long productId);
}
@crichardson
public class WishlistService {
private UserService userService;
private ProductInfoService productInfoService;
public Wishlist getWishlistDetails(long userId) throws Exception {
Future<UserProfile> userProfileFuture = userService.getUserProfile(userId);
UserProfile userProfile = userProfileFuture.get(300, TimeUnit.MILLISECONDS);
Example wish list service
get user
info
List<Future<ProductInfo>> productInfoFutures =
userProfile.getWishListProductIds().stream()
.map(productInfoService::getProductInfo)
.collect(Collectors.toList());
long deadline = System.currentTimeMillis() + 300;
List<ProductInfo> products = new ArrayList<ProductInfo>();
for (Future<ProductInfo> pif : productInfoFutures) {
long timeout = deadline - System.currentTimeMillis();
if (timeout <= 0) throw new TimeoutException(...);
products.add(pif.get(timeout, TimeUnit.MILLISECONDS));
}
...
return new Wishlist(products);
}
asynchronously
get all products
wait for
product
info
@crichardson
It works BUT
Code is very low-level and
messy
And, it’s blocking
@crichardson
Better: Futures with callbacks
no blocking!
def asyncSquare(x : Int)
: Future[Int] = ... x * x...
val f = asyncSquare(25)
Guava ListenableFutures, Spring 4 ListenableFuture
Java 8 CompletableFuture, Scala Futures
f onSuccess {
case x : Int => println(x)
}
f onFailure {
case e : Exception => println("exception thrown")
}
Partial function applied to
successful outcome
Applied to failed outcome
@crichardson
But
callback-based scatter/gather
Messy, tangled code
(aka. callback hell)
@crichardson
Functional futures - map
def asyncPlus(x : Int, y : Int) = ... x + y ...
val future2 = asyncPlus(4, 5).map{ _ * 3 }
assertEquals(27, Await.result(future2, 1 second))
Scala, Java 8 CompletableFuture
Asynchronously transforms
future
@crichardson
Functional futures - flatMap()
val f2 = asyncPlus(5, 8).flatMap { x => asyncSquare(x) }
assertEquals(169, Await.result(f2, 1 second))
Scala, Java 8 CompletableFuture (partially)
Calls asyncSquare() with the eventual
outcome of asyncPlus()
@crichardson
flatMap() is asynchronous
Outcome3f3
Outcome3
f2
f2 = f1 flatMap (someFn)
Outcome1
f1
Implemented using callbacks
someFn(outcome1)
@crichardson
class WishListService(...) {
def getWishList(userId : Long) : Future[WishList] = {
userService.getUserProfile(userId) flatMap { userProfile =>
Scala wishlist service
val futureOfProductsList : Future[List[ProductInfo]] =
Future.sequence(listOfProductFutures)
val timeoutFuture = ...
Future.firstCompletedOf(Seq(wishlist, timeoutFuture))
}
}
val wishlist =
futureOfProductsList.map { products =>
WishList(products) }
val listOfProductFutures : List[Future[ProductInfo]] =
userProfile.wishListProductIds
.map { productInfoService.getProductInfo }
@crichardson
Using Java 8 CompletableFutures
public class UserServiceImpl implements UserService {
@Override
public CompletableFuture<UserInfo> getUserInfo(long userId) {
return CompletableFuture.supplyAsync(
() -> httpGetRequest("http://myuservice/user" + userId,
UserInfo.class));
}
Runs in ExecutorService
@crichardson
Using Java 8 CompletableFutures
public CompletableFuture<Wishlist> getWishlistDetails(long userId) {
return userService.getUserProfile(userId).thenComposeAsync(userProfile -> {
Stream<CompletableFuture<ProductInfo>> s1 =
userProfile.getWishListProductIds()
.stream()
.map(productInfoService::getProductInfo);
Stream<CompletableFuture<List<ProductInfo>>> s2 =
s1.map(fOfPi -> fOfPi.thenApplyAsync(pi -> Arrays.asList(pi)));
CompletableFuture<List<ProductInfo>> productInfos = s2
.reduce((f1, f2) -> f1.thenCombine(f2, ListUtils::union))
.orElse(CompletableFuture.completedFuture(Collections.emptyList()));
return productInfos.thenApply(list -> new Wishlist());
});
}
Java 8 is missing Future.sequence()
flatMap()!
map()!
@crichardson
Introducing Reactive
Extensions (Rx)
The Reactive Extensions (Rx) is a library for composing
asynchronous and event-based programs using
observable sequences and LINQ-style query operators.
Using Rx, developers represent asynchronous data
streams with Observables , query asynchronous
data streams using LINQ operators , and .....
https://rx.codeplex.com/
@crichardson
About RxJava
Reactive Extensions (Rx) for the JVM
Original motivation for Netflix was to provide rich Futures
Implemented in Java
Adaptors for Scala, Groovy and Clojure
https://github.com/Netflix/RxJava
@crichardson
RxJava core concepts
trait Observable[T] {
def subscribe(observer : Observer[T]) : Subscription
...
}
trait Observer[T] {
def onNext(value : T)
def onCompleted()
def onError(e : Throwable)
}
Notifies
An asynchronous stream of items
Used to
unsubscribe
Comparing Observable to...
Observer pattern - similar but
adds
Observer.onComplete()
Observer.onError()
Iterator pattern - mirror image
Push rather than pull
Futures - similar
Can be used as Futures
But Observables = a stream
of multiple values
Collections and Streams -
similar
Functional API supporting
map(), flatMap(), ...
But Observables are
asynchronous
@crichardson
Fun with observables
val every10Seconds = Observable.interval(10 seconds)
-1 0 1 ...
t=0 t=10 t=20 ...
val oneItem = Observable.items(-1L)
val ticker = oneItem ++ every10Seconds
val subscription = ticker.subscribe { (value: Long) => println("value=" + value) }
...
subscription.unsubscribe()
@crichardson
def getTableStatus(tableName: String) : Observable[DynamoDbStatus]=
Observable { subscriber: Subscriber[DynamoDbMessage] =>
}
Connecting observables to the
outside world
amazonDynamoDBAsyncClient.describeTableAsync(
new DescribeTableRequest(tableName),
new AsyncHandler[DescribeTableRequest, DescribeTableResult] {
override def onSuccess(request: DescribeTableRequest,
result: DescribeTableResult) = {
subscriber.onNext(DynamoDbStatus(result.getTable.getTableStatus))
subscriber.onCompleted()
}
override def onError(exception: Exception) = exception match {
case t: ResourceNotFoundException =>
subscriber.onNext(DynamoDbStatus("NOT_FOUND"))
subscriber.onCompleted()
case _ =>
subscriber.onError(exception)
}
})
}
@crichardson
Transforming observables
val tableStatus = ticker.flatMap { i =>
logger.info("{}th describe table", i + 1)
getTableStatus(name)
}
Status1 Status2 Status3 ...
t=0 t=10 t=20 ...
+ Usual collection methods: map(), filter(), take(), drop(), ...
@crichardson
Calculating rolling average
class AverageTradePriceCalculator {
def calculateAverages(trades: Observable[Trade]):
Observable[AveragePrice] = {
...
}
case class Trade(
symbol : String,
price : Double,
quantity : Int
...
)
case class AveragePrice(
symbol : String,
price : Double,
...)
@crichardson
Calculating average pricesdef calculateAverages(trades: Observable[Trade]): Observable[AveragePrice] = {
trades.groupBy(_.symbol).map { symbolAndTrades =>
val (symbol, tradesForSymbol) = symbolAndTrades
val openingEverySecond =
Observable.items(-1L) ++ Observable.interval(1 seconds)
def closingAfterSixSeconds(opening: Any) =
Observable.interval(6 seconds).take(1)
tradesForSymbol.window(...).map {
windowOfTradesForSymbol =>
windowOfTradesForSymbol.fold((0.0, 0, List[Double]())) { (soFar, trade) =>
val (sum, count, prices) = soFar
(sum + trade.price, count + trade.quantity, trade.price +: prices)
} map { x =>
val (sum, length, prices) = x
AveragePrice(symbol, sum / length, prices)
}
}.flatten
}.flatten
}
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
Let’s imagine that you want
to count word frequencies
@crichardson
Scala Word Count
val frequency : Map[String, Int] =
Source.fromFile("gettysburgaddress.txt").getLines()
.flatMap { _.split(" ") }.toList
frequency("THE") should be(11)
frequency("LIBERTY") should be(1)
.groupBy(identity)
.mapValues(_.length))
Map
Reduce
@crichardson
But how to scale to a cluster
of machines?
@crichardson
Apache Hadoop
Open-source software for reliable, scalable, distributed computing
Hadoop Distributed File System (HDFS)
Efficiently stores very large amounts of data
Files are partitioned and replicated across multiple machines
Hadoop MapReduce
Batch processing system
Provides plumbing for writing distributed jobs
Handles failures
...
@crichardson
Overview of MapReduce
Input
Data
Mapper
Mapper
Mapper
Reducer
Reducer
Reducer
Out
put
Data
Shuffle
(K,V)
(K,V)
(K,V)
(K,V)*
(K,V)*
(K,V)*
(K1,V, ....)*
(K2,V, ....)*
(K3,V, ....)*
(K,V)
(K,V)
(K,V)
@crichardson
MapReduce Word count -
mapperclass Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}
(“Four”, 1), (“score”, 1), (“and”, 1), (“seven”, 1), ...
Four score and seven years
http://wiki.apache.org/hadoop/WordCount
@crichardson
Hadoop then shuffles the
key-value pairs...
@crichardson
MapReduce Word count -
reducer
class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key,
Iterable<IntWritable> values, Context context) {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
(“the”, 11)
(“the”, (1, 1, 1, 1, 1, 1, ...))
http://wiki.apache.org/hadoop/WordCount
@crichardson
About MapReduce
Very simple programming abstract yet incredibly powerful
By chaining together multiple map/reduce jobs you can process
very large amounts of data
e.g. Apache Mahout for machine learning
But
Mappers and Reducers = verbose code
Development is challenging, e.g. unit testing is difficult
It’s disk-based, batch processing slow
@crichardson
Scalding: Scala DSL for
MapReduce
class WordCountJob(args : Args) extends Job(args) {
TextLine( args("input") )
.flatMap('line -> 'word) { line : String => tokenize(line) }
.groupBy('word) { _.size }
.write( Tsv( args("output") ) )
def tokenize(text : String) : Array[String] = {
text.toLowerCase.replaceAll("[^a-zA-Z0-9s]", "")
.split("s+")
}
}
https://github.com/twitter/scalding
Expressive and unit testable
Each row is a map of
named fields
@crichardson
Apache Spark
Part of the Hadoop ecosystem
Key abstraction = Resilient Distributed Datasets (RDD)
Collection that is partitioned across cluster members
Operations are parallelized
Created from either a Scala collection or a Hadoop
supported datasource - HDFS, S3 etc
Can be cached in-memory for super-fast performance
Can be replicated for fault-tolerance
http://spark.apache.org
@crichardson
Spark Word Count
val sc = new SparkContext(...)
sc.textFile(“s3n://mybucket/...”)
.flatMap { _.split(" ")}
.groupBy(identity)
.mapValues(_.length)
.toArray.toMap
}
}
Expressive, unit testable and very fast
@crichardson
Summary
Functional programming enables the elegant expression of
good ideas in a wide variety of domains
map(), flatMap() and reduce() are remarkably versatile
higher-order functions
Use FP and OOP together
Java 8 has taken a good first step towards supporting FP
@crichardson
Questions?
@crichardson chris@chrisrichardson.net
http://plainoldobjects.com

Contenu connexe

Tendances

The Power of Composition
The Power of CompositionThe Power of Composition
The Power of CompositionScott Wlaschin
 
Domain Driven Design with the F# type System -- NDC London 2013
Domain Driven Design with the F# type System -- NDC London 2013Domain Driven Design with the F# type System -- NDC London 2013
Domain Driven Design with the F# type System -- NDC London 2013Scott Wlaschin
 
Domain Modeling in a Functional World
Domain Modeling in a Functional WorldDomain Modeling in a Functional World
Domain Modeling in a Functional WorldDebasish Ghosh
 
ZIO-Direct - Functional Scala 2022
ZIO-Direct - Functional Scala 2022ZIO-Direct - Functional Scala 2022
ZIO-Direct - Functional Scala 2022Alexander Ioffe
 
Sequence and Traverse - Part 2
Sequence and Traverse - Part 2Sequence and Traverse - Part 2
Sequence and Traverse - Part 2Philip Schwarz
 
Monoids - Part 1 - with examples using Scalaz and Cats
Monoids - Part 1 - with examples using Scalaz and CatsMonoids - Part 1 - with examples using Scalaz and Cats
Monoids - Part 1 - with examples using Scalaz and CatsPhilip Schwarz
 
Exploring ZIO Prelude: The game changer for typeclasses in Scala
Exploring ZIO Prelude: The game changer for typeclasses in ScalaExploring ZIO Prelude: The game changer for typeclasses in Scala
Exploring ZIO Prelude: The game changer for typeclasses in ScalaJorge Vásquez
 
The lazy programmer's guide to writing thousands of tests
The lazy programmer's guide to writing thousands of testsThe lazy programmer's guide to writing thousands of tests
The lazy programmer's guide to writing thousands of testsScott Wlaschin
 
The Functional Programming Toolkit (NDC Oslo 2019)
The Functional Programming Toolkit (NDC Oslo 2019)The Functional Programming Toolkit (NDC Oslo 2019)
The Functional Programming Toolkit (NDC Oslo 2019)Scott Wlaschin
 
Functional Programming Patterns (NDC London 2014)
Functional Programming Patterns (NDC London 2014)Functional Programming Patterns (NDC London 2014)
Functional Programming Patterns (NDC London 2014)Scott Wlaschin
 
The Functional Programmer's Toolkit (NDC London 2019)
The Functional Programmer's Toolkit (NDC London 2019)The Functional Programmer's Toolkit (NDC London 2019)
The Functional Programmer's Toolkit (NDC London 2019)Scott Wlaschin
 
API Asynchrones en Java 8
API Asynchrones en Java 8API Asynchrones en Java 8
API Asynchrones en Java 8José Paumard
 
Be Smart, Constrain Your Types to Free Your Brain!
Be Smart, Constrain Your Types to Free Your Brain!Be Smart, Constrain Your Types to Free Your Brain!
Be Smart, Constrain Your Types to Free Your Brain!Jorge Vásquez
 
Idiomatic Kotlin
Idiomatic KotlinIdiomatic Kotlin
Idiomatic Kotlinintelliyole
 
Quill vs Slick Smackdown
Quill vs Slick SmackdownQuill vs Slick Smackdown
Quill vs Slick SmackdownAlexander Ioffe
 
Pipeline oriented programming
Pipeline oriented programmingPipeline oriented programming
Pipeline oriented programmingScott Wlaschin
 

Tendances (20)

The Power of Composition
The Power of CompositionThe Power of Composition
The Power of Composition
 
Zio in real world
Zio in real worldZio in real world
Zio in real world
 
Domain Driven Design with the F# type System -- NDC London 2013
Domain Driven Design with the F# type System -- NDC London 2013Domain Driven Design with the F# type System -- NDC London 2013
Domain Driven Design with the F# type System -- NDC London 2013
 
Domain Modeling in a Functional World
Domain Modeling in a Functional WorldDomain Modeling in a Functional World
Domain Modeling in a Functional World
 
ZIO-Direct - Functional Scala 2022
ZIO-Direct - Functional Scala 2022ZIO-Direct - Functional Scala 2022
ZIO-Direct - Functional Scala 2022
 
Sequence and Traverse - Part 2
Sequence and Traverse - Part 2Sequence and Traverse - Part 2
Sequence and Traverse - Part 2
 
Monoids - Part 1 - with examples using Scalaz and Cats
Monoids - Part 1 - with examples using Scalaz and CatsMonoids - Part 1 - with examples using Scalaz and Cats
Monoids - Part 1 - with examples using Scalaz and Cats
 
Exploring ZIO Prelude: The game changer for typeclasses in Scala
Exploring ZIO Prelude: The game changer for typeclasses in ScalaExploring ZIO Prelude: The game changer for typeclasses in Scala
Exploring ZIO Prelude: The game changer for typeclasses in Scala
 
The lazy programmer's guide to writing thousands of tests
The lazy programmer's guide to writing thousands of testsThe lazy programmer's guide to writing thousands of tests
The lazy programmer's guide to writing thousands of tests
 
The Functional Programming Toolkit (NDC Oslo 2019)
The Functional Programming Toolkit (NDC Oslo 2019)The Functional Programming Toolkit (NDC Oslo 2019)
The Functional Programming Toolkit (NDC Oslo 2019)
 
Functional Programming Patterns (NDC London 2014)
Functional Programming Patterns (NDC London 2014)Functional Programming Patterns (NDC London 2014)
Functional Programming Patterns (NDC London 2014)
 
The Functional Programmer's Toolkit (NDC London 2019)
The Functional Programmer's Toolkit (NDC London 2019)The Functional Programmer's Toolkit (NDC London 2019)
The Functional Programmer's Toolkit (NDC London 2019)
 
API Asynchrones en Java 8
API Asynchrones en Java 8API Asynchrones en Java 8
API Asynchrones en Java 8
 
Scala Intro
Scala IntroScala Intro
Scala Intro
 
Be Smart, Constrain Your Types to Free Your Brain!
Be Smart, Constrain Your Types to Free Your Brain!Be Smart, Constrain Your Types to Free Your Brain!
Be Smart, Constrain Your Types to Free Your Brain!
 
Idiomatic Kotlin
Idiomatic KotlinIdiomatic Kotlin
Idiomatic Kotlin
 
Quill vs Slick Smackdown
Quill vs Slick SmackdownQuill vs Slick Smackdown
Quill vs Slick Smackdown
 
Clean coding-practices
Clean coding-practicesClean coding-practices
Clean coding-practices
 
Broadleaf Presents Thymeleaf
Broadleaf Presents ThymeleafBroadleaf Presents Thymeleaf
Broadleaf Presents Thymeleaf
 
Pipeline oriented programming
Pipeline oriented programmingPipeline oriented programming
Pipeline oriented programming
 

En vedette

Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...Chris Richardson
 
Why we cannot ignore Functional Programming
Why we cannot ignore Functional ProgrammingWhy we cannot ignore Functional Programming
Why we cannot ignore Functional ProgrammingMario Fusco
 
OOP and FP - Become a Better Programmer
OOP and FP - Become a Better ProgrammerOOP and FP - Become a Better Programmer
OOP and FP - Become a Better ProgrammerMario Fusco
 
Reactive Programming for a demanding world: building event-driven and respons...
Reactive Programming for a demanding world: building event-driven and respons...Reactive Programming for a demanding world: building event-driven and respons...
Reactive Programming for a demanding world: building event-driven and respons...Mario Fusco
 
Developing microservices with aggregates (devnexus2017)
Developing microservices with aggregates (devnexus2017)Developing microservices with aggregates (devnexus2017)
Developing microservices with aggregates (devnexus2017)Chris Richardson
 
WebCamp:Front-end Developers Day. Александр Мостовенко "Rx.js - делаем асинхр...
WebCamp:Front-end Developers Day. Александр Мостовенко "Rx.js - делаем асинхр...WebCamp:Front-end Developers Day. Александр Мостовенко "Rx.js - делаем асинхр...
WebCamp:Front-end Developers Day. Александр Мостовенко "Rx.js - делаем асинхр...GeeksLab Odessa
 
GKAC 2014 Nov. - 그루비로 안드로이드 앱 개발하기
GKAC 2014 Nov. - 그루비로 안드로이드 앱 개발하기GKAC 2014 Nov. - 그루비로 안드로이드 앱 개발하기
GKAC 2014 Nov. - 그루비로 안드로이드 앱 개발하기GDG Korea
 
Solit 2014, Реактивный Javascript. Победа над асинхронностью и вложенностью, ...
Solit 2014, Реактивный Javascript. Победа над асинхронностью и вложенностью, ...Solit 2014, Реактивный Javascript. Победа над асинхронностью и вложенностью, ...
Solit 2014, Реактивный Javascript. Победа над асинхронностью и вложенностью, ...solit
 
Decomposing applications for deployability and scalability (SpringOne China 2...
Decomposing applications for deployability and scalability (SpringOne China 2...Decomposing applications for deployability and scalability (SpringOne China 2...
Decomposing applications for deployability and scalability (SpringOne China 2...Chris Richardson
 
Polygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @OakjugPolygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @OakjugChris Richardson
 
Developing polyglot persistence applications (devnexus 2013)
Developing polyglot persistence applications (devnexus 2013)Developing polyglot persistence applications (devnexus 2013)
Developing polyglot persistence applications (devnexus 2013)Chris Richardson
 
Developing applications with Cloud Services (Devnexus 2013)
Developing applications with Cloud Services (Devnexus 2013)Developing applications with Cloud Services (Devnexus 2013)
Developing applications with Cloud Services (Devnexus 2013)Chris Richardson
 
Developing applications with Cloud Services #javaone 2012
Developing applications with Cloud Services  #javaone 2012Developing applications with Cloud Services  #javaone 2012
Developing applications with Cloud Services #javaone 2012Chris Richardson
 
Developing polyglot applications on Cloud Foundry (#oredev 2012)
Developing polyglot applications on Cloud Foundry (#oredev 2012)Developing polyglot applications on Cloud Foundry (#oredev 2012)
Developing polyglot applications on Cloud Foundry (#oredev 2012)Chris Richardson
 
Developing polyglot persistence applications #javaone 2012
Developing polyglot persistence applications  #javaone 2012Developing polyglot persistence applications  #javaone 2012
Developing polyglot persistence applications #javaone 2012Chris Richardson
 
Decomposing applications for scalability and deployability - svcc sv_code_ca...
Decomposing applications for scalability and deployability  - svcc sv_code_ca...Decomposing applications for scalability and deployability  - svcc sv_code_ca...
Decomposing applications for scalability and deployability - svcc sv_code_ca...Chris Richardson
 
Developing polyglot persistence applications (SpringOne India 2012)
Developing polyglot persistence applications (SpringOne India 2012)Developing polyglot persistence applications (SpringOne India 2012)
Developing polyglot persistence applications (SpringOne India 2012)Chris Richardson
 
Jumping-with-java8
Jumping-with-java8Jumping-with-java8
Jumping-with-java8Dhaval Dalal
 
Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)Chris Richardson
 

En vedette (20)

Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
 
Carta perú sabor
Carta perú saborCarta perú sabor
Carta perú sabor
 
Why we cannot ignore Functional Programming
Why we cannot ignore Functional ProgrammingWhy we cannot ignore Functional Programming
Why we cannot ignore Functional Programming
 
OOP and FP - Become a Better Programmer
OOP and FP - Become a Better ProgrammerOOP and FP - Become a Better Programmer
OOP and FP - Become a Better Programmer
 
Reactive Programming for a demanding world: building event-driven and respons...
Reactive Programming for a demanding world: building event-driven and respons...Reactive Programming for a demanding world: building event-driven and respons...
Reactive Programming for a demanding world: building event-driven and respons...
 
Developing microservices with aggregates (devnexus2017)
Developing microservices with aggregates (devnexus2017)Developing microservices with aggregates (devnexus2017)
Developing microservices with aggregates (devnexus2017)
 
WebCamp:Front-end Developers Day. Александр Мостовенко "Rx.js - делаем асинхр...
WebCamp:Front-end Developers Day. Александр Мостовенко "Rx.js - делаем асинхр...WebCamp:Front-end Developers Day. Александр Мостовенко "Rx.js - делаем асинхр...
WebCamp:Front-end Developers Day. Александр Мостовенко "Rx.js - делаем асинхр...
 
GKAC 2014 Nov. - 그루비로 안드로이드 앱 개발하기
GKAC 2014 Nov. - 그루비로 안드로이드 앱 개발하기GKAC 2014 Nov. - 그루비로 안드로이드 앱 개발하기
GKAC 2014 Nov. - 그루비로 안드로이드 앱 개발하기
 
Solit 2014, Реактивный Javascript. Победа над асинхронностью и вложенностью, ...
Solit 2014, Реактивный Javascript. Победа над асинхронностью и вложенностью, ...Solit 2014, Реактивный Javascript. Победа над асинхронностью и вложенностью, ...
Solit 2014, Реактивный Javascript. Победа над асинхронностью и вложенностью, ...
 
Decomposing applications for deployability and scalability (SpringOne China 2...
Decomposing applications for deployability and scalability (SpringOne China 2...Decomposing applications for deployability and scalability (SpringOne China 2...
Decomposing applications for deployability and scalability (SpringOne China 2...
 
Polygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @OakjugPolygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @Oakjug
 
Developing polyglot persistence applications (devnexus 2013)
Developing polyglot persistence applications (devnexus 2013)Developing polyglot persistence applications (devnexus 2013)
Developing polyglot persistence applications (devnexus 2013)
 
Developing applications with Cloud Services (Devnexus 2013)
Developing applications with Cloud Services (Devnexus 2013)Developing applications with Cloud Services (Devnexus 2013)
Developing applications with Cloud Services (Devnexus 2013)
 
Developing applications with Cloud Services #javaone 2012
Developing applications with Cloud Services  #javaone 2012Developing applications with Cloud Services  #javaone 2012
Developing applications with Cloud Services #javaone 2012
 
Developing polyglot applications on Cloud Foundry (#oredev 2012)
Developing polyglot applications on Cloud Foundry (#oredev 2012)Developing polyglot applications on Cloud Foundry (#oredev 2012)
Developing polyglot applications on Cloud Foundry (#oredev 2012)
 
Developing polyglot persistence applications #javaone 2012
Developing polyglot persistence applications  #javaone 2012Developing polyglot persistence applications  #javaone 2012
Developing polyglot persistence applications #javaone 2012
 
Decomposing applications for scalability and deployability - svcc sv_code_ca...
Decomposing applications for scalability and deployability  - svcc sv_code_ca...Decomposing applications for scalability and deployability  - svcc sv_code_ca...
Decomposing applications for scalability and deployability - svcc sv_code_ca...
 
Developing polyglot persistence applications (SpringOne India 2012)
Developing polyglot persistence applications (SpringOne India 2012)Developing polyglot persistence applications (SpringOne India 2012)
Developing polyglot persistence applications (SpringOne India 2012)
 
Jumping-with-java8
Jumping-with-java8Jumping-with-java8
Jumping-with-java8
 
Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)
 

Similaire à MAP, FILTER, REDUCE: Functional programming simplifies collections

Map, flatmap and reduce are your new best friends (javaone, svcc)
Map, flatmap and reduce are your new best friends (javaone, svcc)Map, flatmap and reduce are your new best friends (javaone, svcc)
Map, flatmap and reduce are your new best friends (javaone, svcc)Chris Richardson
 
Transformations and actions a visual guide training
Transformations and actions a visual guide trainingTransformations and actions a visual guide training
Transformations and actions a visual guide trainingSpark Summit
 
R for Pythonistas (PyData NYC 2017)
R for Pythonistas (PyData NYC 2017)R for Pythonistas (PyData NYC 2017)
R for Pythonistas (PyData NYC 2017)Christopher Roach
 
NodeJS: the good parts? A skeptic’s view (jax jax2013)
NodeJS: the good parts? A skeptic’s view (jax jax2013)NodeJS: the good parts? A skeptic’s view (jax jax2013)
NodeJS: the good parts? A skeptic’s view (jax jax2013)Chris Richardson
 
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017StampedeCon
 
How to deal with nested lists in R?
How to deal with nested lists in R? How to deal with nested lists in R?
How to deal with nested lists in R? Sotrender
 
Poetry with R -- Dissecting the code
Poetry with R -- Dissecting the codePoetry with R -- Dissecting the code
Poetry with R -- Dissecting the codePeter Solymos
 
SparkR - Play Spark Using R (20160909 HadoopCon)
SparkR - Play Spark Using R (20160909 HadoopCon)SparkR - Play Spark Using R (20160909 HadoopCon)
SparkR - Play Spark Using R (20160909 HadoopCon)wqchen
 
Kotlin Developer Starter in Android projects
Kotlin Developer Starter in Android projectsKotlin Developer Starter in Android projects
Kotlin Developer Starter in Android projectsBartosz Kosarzycki
 
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016STX Next
 
Going reactive in java
Going reactive in javaGoing reactive in java
Going reactive in javaJosé Paumard
 
Mastering Hadoop Map Reduce - Custom Types and Other Optimizations
Mastering Hadoop Map Reduce - Custom Types and Other OptimizationsMastering Hadoop Map Reduce - Custom Types and Other Optimizations
Mastering Hadoop Map Reduce - Custom Types and Other Optimizationsscottcrespo
 
All I know about rsc.io/c2go
All I know about rsc.io/c2goAll I know about rsc.io/c2go
All I know about rsc.io/c2goMoriyoshi Koizumi
 
Soft Shake Event / A soft introduction to Neo4J
Soft Shake Event / A soft introduction to Neo4JSoft Shake Event / A soft introduction to Neo4J
Soft Shake Event / A soft introduction to Neo4JFlorent Biville
 

Similaire à MAP, FILTER, REDUCE: Functional programming simplifies collections (20)

Map, flatmap and reduce are your new best friends (javaone, svcc)
Map, flatmap and reduce are your new best friends (javaone, svcc)Map, flatmap and reduce are your new best friends (javaone, svcc)
Map, flatmap and reduce are your new best friends (javaone, svcc)
 
Transformations and actions a visual guide training
Transformations and actions a visual guide trainingTransformations and actions a visual guide training
Transformations and actions a visual guide training
 
R for Pythonistas (PyData NYC 2017)
R for Pythonistas (PyData NYC 2017)R for Pythonistas (PyData NYC 2017)
R for Pythonistas (PyData NYC 2017)
 
NodeJS: the good parts? A skeptic’s view (jax jax2013)
NodeJS: the good parts? A skeptic’s view (jax jax2013)NodeJS: the good parts? A skeptic’s view (jax jax2013)
NodeJS: the good parts? A skeptic’s view (jax jax2013)
 
What's New In C# 7
What's New In C# 7What's New In C# 7
What's New In C# 7
 
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
 
Cpp tutorial
Cpp tutorialCpp tutorial
Cpp tutorial
 
CppTutorial.ppt
CppTutorial.pptCppTutorial.ppt
CppTutorial.ppt
 
C++ oop
C++ oopC++ oop
C++ oop
 
How to deal with nested lists in R?
How to deal with nested lists in R? How to deal with nested lists in R?
How to deal with nested lists in R?
 
Poetry with R -- Dissecting the code
Poetry with R -- Dissecting the codePoetry with R -- Dissecting the code
Poetry with R -- Dissecting the code
 
SparkR - Play Spark Using R (20160909 HadoopCon)
SparkR - Play Spark Using R (20160909 HadoopCon)SparkR - Play Spark Using R (20160909 HadoopCon)
SparkR - Play Spark Using R (20160909 HadoopCon)
 
Spark - Philly JUG
Spark  - Philly JUGSpark  - Philly JUG
Spark - Philly JUG
 
Kotlin Developer Starter in Android projects
Kotlin Developer Starter in Android projectsKotlin Developer Starter in Android projects
Kotlin Developer Starter in Android projects
 
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
 
Going reactive in java
Going reactive in javaGoing reactive in java
Going reactive in java
 
Mastering Hadoop Map Reduce - Custom Types and Other Optimizations
Mastering Hadoop Map Reduce - Custom Types and Other OptimizationsMastering Hadoop Map Reduce - Custom Types and Other Optimizations
Mastering Hadoop Map Reduce - Custom Types and Other Optimizations
 
Data Analysis in Python
Data Analysis in PythonData Analysis in Python
Data Analysis in Python
 
All I know about rsc.io/c2go
All I know about rsc.io/c2goAll I know about rsc.io/c2go
All I know about rsc.io/c2go
 
Soft Shake Event / A soft introduction to Neo4J
Soft Shake Event / A soft introduction to Neo4JSoft Shake Event / A soft introduction to Neo4J
Soft Shake Event / A soft introduction to Neo4J
 

Plus de Chris Richardson

The microservice architecture: what, why, when and how?
The microservice architecture: what, why, when and how?The microservice architecture: what, why, when and how?
The microservice architecture: what, why, when and how?Chris Richardson
 
More the merrier: a microservices anti-pattern
More the merrier: a microservices anti-patternMore the merrier: a microservices anti-pattern
More the merrier: a microservices anti-patternChris Richardson
 
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...Chris Richardson
 
Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!Chris Richardson
 
Dark energy, dark matter and microservice architecture collaboration patterns
Dark energy, dark matter and microservice architecture collaboration patternsDark energy, dark matter and microservice architecture collaboration patterns
Dark energy, dark matter and microservice architecture collaboration patternsChris Richardson
 
Scenarios_and_Architecture_SkillsMatter_April_2022.pdf
Scenarios_and_Architecture_SkillsMatter_April_2022.pdfScenarios_and_Architecture_SkillsMatter_April_2022.pdf
Scenarios_and_Architecture_SkillsMatter_April_2022.pdfChris Richardson
 
Using patterns and pattern languages to make better architectural decisions
Using patterns and pattern languages to make better architectural decisions Using patterns and pattern languages to make better architectural decisions
Using patterns and pattern languages to make better architectural decisions Chris Richardson
 
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...Chris Richardson
 
Events to the rescue: solving distributed data problems in a microservice arc...
Events to the rescue: solving distributed data problems in a microservice arc...Events to the rescue: solving distributed data problems in a microservice arc...
Events to the rescue: solving distributed data problems in a microservice arc...Chris Richardson
 
A pattern language for microservices - June 2021
A pattern language for microservices - June 2021 A pattern language for microservices - June 2021
A pattern language for microservices - June 2021 Chris Richardson
 
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureQConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureChris Richardson
 
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...Chris Richardson
 
Designing loosely coupled services
Designing loosely coupled servicesDesigning loosely coupled services
Designing loosely coupled servicesChris Richardson
 
Microservices - an architecture that enables DevOps (T Systems DevOps day)
Microservices - an architecture that enables DevOps (T Systems DevOps day)Microservices - an architecture that enables DevOps (T Systems DevOps day)
Microservices - an architecture that enables DevOps (T Systems DevOps day)Chris Richardson
 
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...Chris Richardson
 
Decompose your monolith: Six principles for refactoring a monolith to microse...
Decompose your monolith: Six principles for refactoring a monolith to microse...Decompose your monolith: Six principles for refactoring a monolith to microse...
Decompose your monolith: Six principles for refactoring a monolith to microse...Chris Richardson
 
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...Chris Richardson
 
Overview of the Eventuate Tram Customers and Orders application
Overview of the Eventuate Tram Customers and Orders applicationOverview of the Eventuate Tram Customers and Orders application
Overview of the Eventuate Tram Customers and Orders applicationChris Richardson
 
An overview of the Eventuate Platform
An overview of the Eventuate PlatformAn overview of the Eventuate Platform
An overview of the Eventuate PlatformChris Richardson
 
#DevNexus202 Decompose your monolith
#DevNexus202 Decompose your monolith#DevNexus202 Decompose your monolith
#DevNexus202 Decompose your monolithChris Richardson
 

Plus de Chris Richardson (20)

The microservice architecture: what, why, when and how?
The microservice architecture: what, why, when and how?The microservice architecture: what, why, when and how?
The microservice architecture: what, why, when and how?
 
More the merrier: a microservices anti-pattern
More the merrier: a microservices anti-patternMore the merrier: a microservices anti-pattern
More the merrier: a microservices anti-pattern
 
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
 
Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!
 
Dark energy, dark matter and microservice architecture collaboration patterns
Dark energy, dark matter and microservice architecture collaboration patternsDark energy, dark matter and microservice architecture collaboration patterns
Dark energy, dark matter and microservice architecture collaboration patterns
 
Scenarios_and_Architecture_SkillsMatter_April_2022.pdf
Scenarios_and_Architecture_SkillsMatter_April_2022.pdfScenarios_and_Architecture_SkillsMatter_April_2022.pdf
Scenarios_and_Architecture_SkillsMatter_April_2022.pdf
 
Using patterns and pattern languages to make better architectural decisions
Using patterns and pattern languages to make better architectural decisions Using patterns and pattern languages to make better architectural decisions
Using patterns and pattern languages to make better architectural decisions
 
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
 
Events to the rescue: solving distributed data problems in a microservice arc...
Events to the rescue: solving distributed data problems in a microservice arc...Events to the rescue: solving distributed data problems in a microservice arc...
Events to the rescue: solving distributed data problems in a microservice arc...
 
A pattern language for microservices - June 2021
A pattern language for microservices - June 2021 A pattern language for microservices - June 2021
A pattern language for microservices - June 2021
 
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureQConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
 
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
 
Designing loosely coupled services
Designing loosely coupled servicesDesigning loosely coupled services
Designing loosely coupled services
 
Microservices - an architecture that enables DevOps (T Systems DevOps day)
Microservices - an architecture that enables DevOps (T Systems DevOps day)Microservices - an architecture that enables DevOps (T Systems DevOps day)
Microservices - an architecture that enables DevOps (T Systems DevOps day)
 
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
 
Decompose your monolith: Six principles for refactoring a monolith to microse...
Decompose your monolith: Six principles for refactoring a monolith to microse...Decompose your monolith: Six principles for refactoring a monolith to microse...
Decompose your monolith: Six principles for refactoring a monolith to microse...
 
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
 
Overview of the Eventuate Tram Customers and Orders application
Overview of the Eventuate Tram Customers and Orders applicationOverview of the Eventuate Tram Customers and Orders application
Overview of the Eventuate Tram Customers and Orders application
 
An overview of the Eventuate Platform
An overview of the Eventuate PlatformAn overview of the Eventuate Platform
An overview of the Eventuate Platform
 
#DevNexus202 Decompose your monolith
#DevNexus202 Decompose your monolith#DevNexus202 Decompose your monolith
#DevNexus202 Decompose your monolith
 

Dernier

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited
 
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 

Dernier (20)

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
 
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 

MAP, FILTER, REDUCE: Functional programming simplifies collections

  • 1. @crichardson Map(), flatMap() and reduce() are your new best friends: Simpler collections, concurrency, and big data Chris Richardson Author of POJOs in Action Founder of the original CloudFoundry.com @crichardson chris@chrisrichardson.net http://plainoldobjects.com
  • 2. @crichardson Presentation goal How functional programming simplifies your code Show that map(), flatMap() and reduce() are remarkably versatile functions
  • 4. @crichardson About Chris Founder of a buzzword compliant (stealthy, social, mobile, big data, machine learning, ...) startup Consultant helping organizations improve how they architect and deploy applications using cloud, micro services, polyglot applications, NoSQL, ...
  • 5. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 8. @crichardson It’s a kind of programming language
  • 9. @crichardson Functions as the building blocks of the application
  • 10. @crichardson Functions as first class citizens Assign functions to variables Store functions in fields Use and write higher-order functions: Pass functions as arguments Return functions as values
  • 11. @crichardson Avoids mutable state Use: Immutable data structures Single assignment variables Some functional languages such as Haskell don’t side-effects There are benefits to immutability Easier concurrency More reliable code But be pragmatic
  • 12. @crichardson Why functional programming? "the highest goal of programming-language design to enable good ideas to be elegantly expressed" http://en.wikipedia.org/wiki/Tony_Hoare
  • 13. @crichardson Why functional programming? More expressive More concise More intuitive - solution matches problem definition Elimination of error-prone mutable state Easy parallelization
  • 14. @crichardson An ancient idea that has recently become popular
  • 16. @crichardson Lisp = an early functional language invented in 1958 http://en.wikipedia.org/wiki/ Lisp_(programming_language) 1940 1950 1960 1970 1980 1990 2000 2010 garbage collection dynamic typing self-hosting compiler tree data structures (defun factorial (n) (if (<= n 1) 1 (* n (factorial (- n 1)))))
  • 17. @crichardson My final year project in 1985: Implementing SASL sieve (p:xs) = p : sieve [x | x <- xs, rem x p > 0]; primes = sieve [2..] A list of integers starting with 2 Filter out multiples of p
  • 18. Mostly an Ivory Tower technology Lisp was used for AI FP languages: Miranda, ML, Haskell, ... “Side-effects kills kittens and puppies”
  • 20. @crichardson But today FP is mainstream Clojure - a dialect of Lisp A hybrid OO/functional language A hybrid OO/FP language for .NET Java 8 has lambda expressions
  • 21. @crichardson Java 8 lambda expressions are functions x -> x * x x -> { for (int i = 2; i < Math.sqrt(x); i = i + 1) { if (x % i == 0) return false; } return true; }; (x, y) -> x * x + y * y
  • 22. @crichardson Java 8 lambdas are a shorthand* for an anonymous inner class * not exactly. See http://programmers.stackexchange.com/questions/ 177879/type-inference-in-java-8
  • 23. @crichardson Java 8 functional interfaces Interface with a single abstract method e.g. Runnable, Callable, Spring’s TransactionCallback A lambda expression is an instance of a functional interface. You can use a lambda wherever a function interface “value” is expected The type of the lambda expression is determined from it’s context
  • 24. @crichardson Example Functional Interface Function<Integer, Integer> square = x -> x * x; BiFunction<Integer, Integer, Integer> sumSquares = (x, y) -> x * x + y * y; Predicate<Integer> makeIsDivisibleBy(int y) { return x -> x % y == 0; } Predicate<Integer> isEven = makeIsDivisibleBy(2); Assert.assertTrue(isEven.test(8)); Assert.assertFalse(isEven.test(11));
  • 25. @crichardson Example Functional Interface ExecutorService executor = ...; final int x = 999 Future<Boolean> outcome = executor.submit(() -> { for (int i = 2; i < Math.sqrt(x); i = i + 1) { if (x % i == 0) return false; } return true; } This lambda is a Callable
  • 26. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 27. @crichardson Lot’s of application code = collection processing: Mapping, filtering, and reducing
  • 28. @crichardson Social network example public class Person { enum Gender { MALE, FEMALE } private Name name; private LocalDate birthday; private Gender gender; private Hometown hometown; private Set<Friend> friends = new HashSet<Friend>(); .... public class Friend { private Person friend; private LocalDate becameFriends; ... } public class SocialNetwork { private Set<Person> people; ...
  • 29. @crichardson Mapping, filtering, and reducing public class Person { public Set<Hometown> hometownsOfFriends() { Set<Hometown> result = new HashSet<>(); for (Friend friend : friends) { result.add(friend.getPerson().getHometown()); } return result; }
  • 30. @crichardson Mapping, filtering, and reducing public class Person { public Set<Person> friendOfFriends() { Set<Person> result = new HashSet(); for (Friend friend : friends) for (Friend friendOfFriend : friend.getPerson().friends) if (friendOfFriend.getPerson() != this) result.add(friendOfFriend.getPerson()); return result; }
  • 31. @crichardson Mapping, filtering, and reducing public class SocialNetwork { private Set<Person> people; ... public Set<Person> lonelyPeople() { Set<Person> result = new HashSet<Person>(); for (Person p : people) { if (p.getFriends().isEmpty()) result.add(p); } return result; }
  • 32. @crichardson Mapping, filtering, and reducing public class SocialNetwork { private Set<Person> people; ... public int averageNumberOfFriends() { int sum = 0; for (Person p : people) { sum += p.getFriends().size(); } return sum / people.size(); }
  • 33. @crichardson Problems with this style of programming Low level Imperative (how to do it) NOT declarative (what to do) Verbose Mutable variables are potentially error prone Difficult to parallelize
  • 34. @crichardson Java 8 streams to the rescue A sequence of elements “Wrapper” around a collection Streams can also be infinite Provides a functional/lambda-based API for transforming, filtering and aggregating elements Much simpler, cleaner code
  • 35. @crichardson Using Java 8 streams - mapping class Person .. private Set<Friend> friends = ...; public Set<Hometown> hometownsOfFriends() { return friends.stream() .map(f -> f.getPerson().getHometown()) .collect(Collectors.toSet()); }
  • 36. @crichardson The map() function s1 a b c d e ... s2 f(a) f(b) f(c) f(d) f(e) ... s2 = s1.map(f)
  • 37. @crichardson public class SocialNetwork { private Set<Person> people; ... public Set<Person> peopleWithNoFriends() { Set<Person> result = new HashSet<Person>(); for (Person p : people) { if (p.getFriends().isEmpty()) result.add(p); } return result; } Using Java 8 streams - filtering public class SocialNetwork { private Set<Person> people; ... public Set<Person> lonelyPeople() { return people.stream() .filter(p -> p.getFriends().isEmpty()) .collect(Collectors.toSet()); }
  • 38. @crichardson Using Java 8 streams - friend of friends V1 class Person .. public Set<Person> friendOfFriends() { Set<Set<Friend>> fof = friends.stream() .map(friend -> friend.getPerson().friends) .collect(Collectors.toSet()); ... } Using map() => Set of Sets :-( Somehow we need to flatten
  • 39. @crichardson Using Java 8 streams - mapping class Person .. public Set<Person> friendOfFriends() { return friends.stream() .flatMap(friend -> friend.getPerson().friends.stream()) .map(Friend::getPerson) .filter(f -> f != this) .collect(Collectors.toSet()); } maps and flattens
  • 40. @crichardson The flatMap() function s1 a b ... s2 f(a)0 f(a)1 f(b)0 f(b)1 f(b)2 ... s2 = s1.flatMap(f)
  • 41. @crichardson Using Java 8 streams - reducing public class SocialNetwork { private Set<Person> people; ... public long averageNumberOfFriends() { return people.stream() .map ( p -> p.getFriends().size() ) .reduce(0, (x, y) -> x + y) / people.size(); } int x = 0; for (int y : inputStream) x = x + y return x;
  • 42. @crichardson The reduce() function s1 a b c d e ... x = s1.reduce(initial, f) f(f(f(f(f(f(initial, a), b), c), d), e), ...)
  • 43. @crichardson Newton's method for finding square roots public class SquareRootCalculator { public double squareRoot(double input, double precision) { return Stream.iterate( new Result(1.0), current -> refine(current, input, precision)) .filter(r -> r.done) .findFirst().get().value; } private static Result refine(Result current, double input, double precision) { double value = current.value; double newCurrent = value - (value * value - input) / (2 * value); boolean done = Math.abs(value - newCurrent) < precision; return new Result(newCurrent, done); } class Result { boolean done; double value; } Creates an infinite stream: seed, f(seed), f(f(seed)), ..... Don’t panic! Streams are lazy
  • 44. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 45. @crichardson Tony’s $1B mistake “I call it my billion-dollar mistake. It was the invention of the null reference in 1965....But I couldn't resist the temptation to put in a null reference, simply because it was so easy to implement...” http://qconlondon.com/london-2009/presentation/ Null+References:+The+Billion+Dollar+Mistake
  • 46. @crichardson Coding with null pointers class Person public Friend longestFriendship() { Friend result = null; for (Friend friend : friends) { if (result == null || friend.getBecameFriends() .isBefore(result.getBecameFriends())) result = friend; } return result; } Friend oldestFriend = person.longestFriendship(); if (oldestFriend != null) { ... } else { ... } Null check is essential yet easily forgotten
  • 47. @crichardson Java 8 Optional<T> A wrapper for nullable references It has two states: empty throws an exception if you try to get the reference non-empty contain a non-null reference Provides methods for: testing whether it has a value getting the value ... Return reference wrapped in an instance of this type instead of null
  • 48. @crichardson Coding with optionals class Person public Optional<Friend> longestFriendship() { Friend result = null; for (Friend friend : friends) { if (result == null || friend.getBecameFriends().isBefore(result.getBecameFriends())) result = friend; } return Optional.ofNullable(result); } Optional<Friend> oldestFriend = person.longestFriendship(); // Might throw java.util.NoSuchElementException: No value present // Person dangerous = popularPerson.get(); if (oldestFriend.isPresent) { ...oldestFriend.get() } else { ... }
  • 49. @crichardson Using Optionals - better Optional<Friend> oldestFriendship = ...; Friend whoToCall1 = oldestFriendship.orElse(mother); Avoid calling isPresent() and get() Friend whoToCall3 = oldestFriendship.orElseThrow( () -> new LonelyPersonException()); Friend whoToCall2 = oldestFriendship.orElseGet(() -> lazilyFindSomeoneElse());
  • 50. @crichardson Using Optional.map() public class Person { public Optional<Friend> longestFriendship() { return ...; } public Optional<Long> ageDifferenceWithOldestFriend() { Optional<Friend> oldestFriend = longestFriendship(); return oldestFriend.map ( of -> Math.abs(of.getPerson().getAge() - getAge())) ); } Eliminates messy conditional logic
  • 51. @crichardson Using flatMap() class Person public Optional<Friend> longestFriendship() {...} public Optional<Friend> longestFriendshipOfLongestFriend() { return longestFriendship() .flatMap(friend -> friend.getPerson().longestFriendship()); } not always a symmetric relationship. :-)
  • 52. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 53. @crichardson Let’s imagine you are performing a CPU intensive operation class Person .. public Set<Hometown> hometownsOfFriends() { return friends.stream() .map(f -> cpuIntensiveOperation()) .collect(Collectors.toSet()); }
  • 54. @crichardson class Person .. public Set<Hometown> hometownsOfFriends() { return friends.parallelStream() .map(f -> cpuIntensiveOperation()) .collect(Collectors.toSet()); } Parallel streams = simple concurrency Potentially uses N cores Nx speed up
  • 55. @crichardson Let’s imagine that you are writing code to display the products in a user’s wish list
  • 56. @crichardson The need for concurrency Step #1 Web service request to get the user profile including wish list (list of product Ids) Step #2 For each productId: web service request to get product info Sequentially terrible response time Need fetch productInfo concurrently
  • 57. @crichardson Futures are a great concurrency abstraction http://en.wikipedia.org/wiki/Futures_and_promises
  • 58. @crichardson Worker thread or event-driven Main thread How futures work Outcome Future Client get Asynchronous operation set initiates
  • 59. @crichardson Benefits Simple way for two concurrent activities to communicate safely Abstraction: Client does not know how the asynchronous operation is implemented Easy to implement scatter/gather: Scatter: Client can invoke multiple asynchronous operations and gets a Future for each one. Gather: Get values from the futures
  • 60. @crichardson Example wish list service public interface UserService { Future<UserProfile> getUserProfile(long userId); } public class UserServiceProxy implements UserService { private ExecutorService executorService; @Override public Future<UserProfile> getUserProfile(long userId) { return executorService.submit(() -> restfulGet("http://uservice/user/" + userId, UserProfile.class)); } ... } public interface ProductInfoService { Future<ProductInfo> getProductInfo(long productId); }
  • 61. @crichardson public class WishlistService { private UserService userService; private ProductInfoService productInfoService; public Wishlist getWishlistDetails(long userId) throws Exception { Future<UserProfile> userProfileFuture = userService.getUserProfile(userId); UserProfile userProfile = userProfileFuture.get(300, TimeUnit.MILLISECONDS); Example wish list service get user info List<Future<ProductInfo>> productInfoFutures = userProfile.getWishListProductIds().stream() .map(productInfoService::getProductInfo) .collect(Collectors.toList()); long deadline = System.currentTimeMillis() + 300; List<ProductInfo> products = new ArrayList<ProductInfo>(); for (Future<ProductInfo> pif : productInfoFutures) { long timeout = deadline - System.currentTimeMillis(); if (timeout <= 0) throw new TimeoutException(...); products.add(pif.get(timeout, TimeUnit.MILLISECONDS)); } ... return new Wishlist(products); } asynchronously get all products wait for product info
  • 62. @crichardson It works BUT Code is very low-level and messy And, it’s blocking
  • 63. @crichardson Better: Futures with callbacks no blocking! def asyncSquare(x : Int) : Future[Int] = ... x * x... val f = asyncSquare(25) Guava ListenableFutures, Spring 4 ListenableFuture Java 8 CompletableFuture, Scala Futures f onSuccess { case x : Int => println(x) } f onFailure { case e : Exception => println("exception thrown") } Partial function applied to successful outcome Applied to failed outcome
  • 65. @crichardson Functional futures - map def asyncPlus(x : Int, y : Int) = ... x + y ... val future2 = asyncPlus(4, 5).map{ _ * 3 } assertEquals(27, Await.result(future2, 1 second)) Scala, Java 8 CompletableFuture Asynchronously transforms future
  • 66. @crichardson Functional futures - flatMap() val f2 = asyncPlus(5, 8).flatMap { x => asyncSquare(x) } assertEquals(169, Await.result(f2, 1 second)) Scala, Java 8 CompletableFuture (partially) Calls asyncSquare() with the eventual outcome of asyncPlus()
  • 67. @crichardson flatMap() is asynchronous Outcome3f3 Outcome3 f2 f2 = f1 flatMap (someFn) Outcome1 f1 Implemented using callbacks someFn(outcome1)
  • 68. @crichardson class WishListService(...) { def getWishList(userId : Long) : Future[WishList] = { userService.getUserProfile(userId) flatMap { userProfile => Scala wishlist service val futureOfProductsList : Future[List[ProductInfo]] = Future.sequence(listOfProductFutures) val timeoutFuture = ... Future.firstCompletedOf(Seq(wishlist, timeoutFuture)) } } val wishlist = futureOfProductsList.map { products => WishList(products) } val listOfProductFutures : List[Future[ProductInfo]] = userProfile.wishListProductIds .map { productInfoService.getProductInfo }
  • 69. @crichardson Using Java 8 CompletableFutures public class UserServiceImpl implements UserService { @Override public CompletableFuture<UserInfo> getUserInfo(long userId) { return CompletableFuture.supplyAsync( () -> httpGetRequest("http://myuservice/user" + userId, UserInfo.class)); } Runs in ExecutorService
  • 70. @crichardson Using Java 8 CompletableFutures public CompletableFuture<Wishlist> getWishlistDetails(long userId) { return userService.getUserProfile(userId).thenComposeAsync(userProfile -> { Stream<CompletableFuture<ProductInfo>> s1 = userProfile.getWishListProductIds() .stream() .map(productInfoService::getProductInfo); Stream<CompletableFuture<List<ProductInfo>>> s2 = s1.map(fOfPi -> fOfPi.thenApplyAsync(pi -> Arrays.asList(pi))); CompletableFuture<List<ProductInfo>> productInfos = s2 .reduce((f1, f2) -> f1.thenCombine(f2, ListUtils::union)) .orElse(CompletableFuture.completedFuture(Collections.emptyList())); return productInfos.thenApply(list -> new Wishlist()); }); } Java 8 is missing Future.sequence() flatMap()! map()!
  • 71. @crichardson Introducing Reactive Extensions (Rx) The Reactive Extensions (Rx) is a library for composing asynchronous and event-based programs using observable sequences and LINQ-style query operators. Using Rx, developers represent asynchronous data streams with Observables , query asynchronous data streams using LINQ operators , and ..... https://rx.codeplex.com/
  • 72. @crichardson About RxJava Reactive Extensions (Rx) for the JVM Original motivation for Netflix was to provide rich Futures Implemented in Java Adaptors for Scala, Groovy and Clojure https://github.com/Netflix/RxJava
  • 73. @crichardson RxJava core concepts trait Observable[T] { def subscribe(observer : Observer[T]) : Subscription ... } trait Observer[T] { def onNext(value : T) def onCompleted() def onError(e : Throwable) } Notifies An asynchronous stream of items Used to unsubscribe
  • 74. Comparing Observable to... Observer pattern - similar but adds Observer.onComplete() Observer.onError() Iterator pattern - mirror image Push rather than pull Futures - similar Can be used as Futures But Observables = a stream of multiple values Collections and Streams - similar Functional API supporting map(), flatMap(), ... But Observables are asynchronous
  • 75. @crichardson Fun with observables val every10Seconds = Observable.interval(10 seconds) -1 0 1 ... t=0 t=10 t=20 ... val oneItem = Observable.items(-1L) val ticker = oneItem ++ every10Seconds val subscription = ticker.subscribe { (value: Long) => println("value=" + value) } ... subscription.unsubscribe()
  • 76. @crichardson def getTableStatus(tableName: String) : Observable[DynamoDbStatus]= Observable { subscriber: Subscriber[DynamoDbMessage] => } Connecting observables to the outside world amazonDynamoDBAsyncClient.describeTableAsync( new DescribeTableRequest(tableName), new AsyncHandler[DescribeTableRequest, DescribeTableResult] { override def onSuccess(request: DescribeTableRequest, result: DescribeTableResult) = { subscriber.onNext(DynamoDbStatus(result.getTable.getTableStatus)) subscriber.onCompleted() } override def onError(exception: Exception) = exception match { case t: ResourceNotFoundException => subscriber.onNext(DynamoDbStatus("NOT_FOUND")) subscriber.onCompleted() case _ => subscriber.onError(exception) } }) }
  • 77. @crichardson Transforming observables val tableStatus = ticker.flatMap { i => logger.info("{}th describe table", i + 1) getTableStatus(name) } Status1 Status2 Status3 ... t=0 t=10 t=20 ... + Usual collection methods: map(), filter(), take(), drop(), ...
  • 78. @crichardson Calculating rolling average class AverageTradePriceCalculator { def calculateAverages(trades: Observable[Trade]): Observable[AveragePrice] = { ... } case class Trade( symbol : String, price : Double, quantity : Int ... ) case class AveragePrice( symbol : String, price : Double, ...)
  • 79. @crichardson Calculating average pricesdef calculateAverages(trades: Observable[Trade]): Observable[AveragePrice] = { trades.groupBy(_.symbol).map { symbolAndTrades => val (symbol, tradesForSymbol) = symbolAndTrades val openingEverySecond = Observable.items(-1L) ++ Observable.interval(1 seconds) def closingAfterSixSeconds(opening: Any) = Observable.interval(6 seconds).take(1) tradesForSymbol.window(...).map { windowOfTradesForSymbol => windowOfTradesForSymbol.fold((0.0, 0, List[Double]())) { (soFar, trade) => val (sum, count, prices) = soFar (sum + trade.price, count + trade.quantity, trade.price +: prices) } map { x => val (sum, length, prices) = x AveragePrice(symbol, sum / length, prices) } }.flatten }.flatten }
  • 80. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 81. @crichardson Let’s imagine that you want to count word frequencies
  • 82. @crichardson Scala Word Count val frequency : Map[String, Int] = Source.fromFile("gettysburgaddress.txt").getLines() .flatMap { _.split(" ") }.toList frequency("THE") should be(11) frequency("LIBERTY") should be(1) .groupBy(identity) .mapValues(_.length)) Map Reduce
  • 83. @crichardson But how to scale to a cluster of machines?
  • 84. @crichardson Apache Hadoop Open-source software for reliable, scalable, distributed computing Hadoop Distributed File System (HDFS) Efficiently stores very large amounts of data Files are partitioned and replicated across multiple machines Hadoop MapReduce Batch processing system Provides plumbing for writing distributed jobs Handles failures ...
  • 86. @crichardson MapReduce Word count - mapperclass Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } (“Four”, 1), (“score”, 1), (“and”, 1), (“seven”, 1), ... Four score and seven years http://wiki.apache.org/hadoop/WordCount
  • 87. @crichardson Hadoop then shuffles the key-value pairs...
  • 88. @crichardson MapReduce Word count - reducer class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } (“the”, 11) (“the”, (1, 1, 1, 1, 1, 1, ...)) http://wiki.apache.org/hadoop/WordCount
  • 89. @crichardson About MapReduce Very simple programming abstract yet incredibly powerful By chaining together multiple map/reduce jobs you can process very large amounts of data e.g. Apache Mahout for machine learning But Mappers and Reducers = verbose code Development is challenging, e.g. unit testing is difficult It’s disk-based, batch processing slow
  • 90. @crichardson Scalding: Scala DSL for MapReduce class WordCountJob(args : Args) extends Job(args) { TextLine( args("input") ) .flatMap('line -> 'word) { line : String => tokenize(line) } .groupBy('word) { _.size } .write( Tsv( args("output") ) ) def tokenize(text : String) : Array[String] = { text.toLowerCase.replaceAll("[^a-zA-Z0-9s]", "") .split("s+") } } https://github.com/twitter/scalding Expressive and unit testable Each row is a map of named fields
  • 91. @crichardson Apache Spark Part of the Hadoop ecosystem Key abstraction = Resilient Distributed Datasets (RDD) Collection that is partitioned across cluster members Operations are parallelized Created from either a Scala collection or a Hadoop supported datasource - HDFS, S3 etc Can be cached in-memory for super-fast performance Can be replicated for fault-tolerance http://spark.apache.org
  • 92. @crichardson Spark Word Count val sc = new SparkContext(...) sc.textFile(“s3n://mybucket/...”) .flatMap { _.split(" ")} .groupBy(identity) .mapValues(_.length) .toArray.toMap } } Expressive, unit testable and very fast
  • 93. @crichardson Summary Functional programming enables the elegant expression of good ideas in a wide variety of domains map(), flatMap() and reduce() are remarkably versatile higher-order functions Use FP and OOP together Java 8 has taken a good first step towards supporting FP