Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
About me:
@AntyaDev
like types*
- Biggest provider in sport offering
- Supports a lot of regulated markets
- About 1000 microservices (200 distinct types)...
highload and
near real time
Why F#?
Why F#?
If we have
sexy C#
If we have sexy C#
class Person
{
public string FirstName; // Not null
public string? MiddleName; // May be null
public st...
If we have sexy C#
switch (shape)
{
case Circle c:
WriteLine($"circle with radius {c.Radius}");
break;
case Square s when ...
If we have sexy C#
public (string, string) LookupName(long id)
{
// some code..
return (first, middle);
}
var names = Look...
Why F#?
If we have
sexy C#
1 day
Story of a new C# project
Get("/api/bets", async request =>
{
var result = Validator.Validate(request); // unit testable
i...
1 day
In 1 week
public class BetsProviderService : IBetsProviderService
{
readonly IOpenBetsRepository _openBetsRepository;
readonly ISett...
Different Culture
OOP FP
abstraction
extensibility
purity
composability
correctness
Triggers:
Much better tooling support
Community growth
Recursive modules
Tooling support
Visual Studio
RiderVisual Studio Code
Recursive modules
type Plane() = member x.Start() = PlaneLogic.start(x)
// types and modules can't currently be mutually
/...
Recursive modules
module M
type First =
| T of Second
and Second =
| T of First
Recursive modules
module M
type First =
| T of Second
type Second =
| T of First
Recursive modules
module rec M
type First =
| T of Second
type Second =
| T of First
Recursive modules
module M
let private helper () = ...
let publicApiFunc () = helper()
Recursive modules
module rec M
let publicApiFunc () = helper()
let private helper () = ...
Community growth
• To begin, F# has grown to be bigger than ever, at least as far as
we can measure, through product telem...
namespace SportData.ETL.Core
type ItemRawUpdate<'Item> = {
Id: string
Data: 'Item option
}
module TransformationFlow =
let...
Code sample
let createGame (globalCache: GlobalItemsCache,
masterEvent: MasterEventItem, currentTime: DateTime) = trial {
...
I need website
Website
SBTech story
I need an API
to build
unique UI
The Problem
Oops?!
The Problem
Oops?!
1. well defined contracts (Event, Market, Selection)
2. flexible way to query data and subscribe on cha...
PUSH based
Queryable API
(Change Feed)
select * from Games where isLive = true
order by totalBets desc
limit 10
PUSH changes
Change Feed
(Actor)
Change Stream
Feed View
2:2
1:0 Chelsea - Milan
Milan - Liverpool
Subscribers
Feed View
2:2
1:0 Chelsea - Milan
Milan - Liverpool
Change Log
Query
Change Feed
(Actor)
Subscribers
type NodeName = string
type LastHeartBeat = DateTime
type NodeSubscribers = Map<NodeName, LastHeartBeat>
type ...
module ChangeFeed
let getSnapshot (feed: ChangeFeed, fastPreloadAmount: uint32) = ...
let getUpdates (feed: ChangeFeed, la...
Feed View
2:2
1:0 Chelsea - Milan
Milan - Liverpool
type PartialView = {
EntityType: EntityType
Entities: IEntity seq
}
ty...
[<Property>]
let ``empty ChangeLog should start with offset 0 and requested maxSize`` (maxSize: uint32) =
let changeLog = ...
We have our own frontend
but
we need a Data
The Next Challenge
Oops?!
SBTech
All changes
Operator
RDBMS
All changes
from
40 tables
ETL
Operator
Well defined entities
compose entities
react on changes
recalculate odds
(da...
C#
F#Asp NET Core
Orleans
DB Drivers
[ DDD; Tests]
Infrastructure
business logic
public Task<FullFeedUpdate> GetSnapshot(GetSnapshot msg)
{
// invoke F# code from C#
var updates = ChangeFeedModule.getSna...
Applied at the heart of the system
Zero Null Reference exceptions
Domain errors instead of exceptions
DDD with types (stro...
http://nbomber.com
- Technology agnostic
(no dependency on HTTP, WebSockets, SSE)
- Very simple API
(to test Pull and Push scenarios)
- CI/CD...
- Technology agnostic
(no dependency on HTTP, WebSockets, SSE)
- Very simple API
(to test Pull and Push scenarios)
- CI/CD...
- Technology agnostic
(no dependency on HTTP, WebSockets, SSE)
- Very simple API
(to test Pull and Push scenarios)
- CI/CD...
- Technology agnostic
(no dependency on HTTP, WebSockets, SSE)
- Very simple API
(to test Pull and Push scenarios)
- CI/CD...
type Step =
| Request of RequestStep // to model Request-response pattern
| Listener of ListenerStep // to model Pub/Sub p...
let execStep (step: Step, req: Request, timer: Stopwatch) = task {
timer.Restart()
match step with
| Request r -> let! res...
type TestFlowRunner(flow: TestFlow) =
let createActors (flow: TestFlow) =
flow.CorrelationIds
|> Set.toArray
|> Array.map(...
// C# example
var flow = new TestFlow(flowName: "READ Users",
steps: new[] {step1, step2, step3},
concurrentCopies: 10);
n...
// F# example
let flow = { FlowName = "GET flow"
Steps = [step1; step2; step3]
ConcurrentCopies = 100 }
Scenario.create("T...
var httpClient = new HttpClient();
var httpStep = Step.CreateRequest("GET request", execute: async _ => {
var response = a...
RDBMS
Storage
Kafka ETL
Kafka
NoSQL
Storage
Push
API
- 1470 LOC
- (4) OOP objects
- (~100) functions
- try/catch used 2 times
async { let! x = myFunction()
match x with
| Some v -> ...
| None -> ... }
F# vNext
match!
async { match! myFunction() with
| Some v -> ...
| None -> ... }
F# vNext
match!
F# vNext
Struct Records
type Vector3 = { X: float; Y: float; Z: float }
[<Struct>]
type Vector3 = { X: float; Y: float; Z:...
F# vNext
Struct Records
[<Struct>]
type Result<'T,'TError> =
| Ok of 'T
| Error of 'Terror
[<Struct>]
type ValueOption<'T>...
F# vNext
Anonymous Records
type Data = {
X: int
Y: string
}
let data = { X = 1; Y = "abc" }
F# vNext
Anonymous Records
let data = {| X = 1; Y = "abc" |}
F# vNext
Structural Type System
type User() =
member this.GetName() = "test user"
type Car() =
member this.GetName() = "te...
F# vNext
Applicative functors
validate {
let! name = validateName(name)
let! age = validateAge(age)
let! email = validateE...
F# vNext
Applicative functors
validate {
let! name = validateName(name)
and! age = validateAge(age)
and! email = validateE...
F# vNext
Applicative functors
parallel {
let! x = slowRequestX()
and! y = slowRequestY()
and! z = slowRequestZ()
return f(...
THANKS
@antyadev
antyadev@gmail.com
@nbombertest
http://nbomber.com
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech
Prochain SlideShare
Chargement dans…5
×

.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech

387 vues

Publié le

В 2017 году мы начали активно использовать F# для построения high-load push-based queryable API, а также обработки больших потоков данных (stateful stream processing). На тот момент времени никто в наших командах не имел предыдущего опыта по внедрению и применению F# но мы решили попробовать. На этом докладе я расскажу о нашем опыте внедрения F#, его проблемах и недостатках, о том как мы его научились готовить, где имеет смысл его применять и как подружить C#/OOP с F#/FP в одном проекте.
Данный доклад нацелен на аудиторию не имеющую предыдущего опыта с FP/F#.
Agenda:
- Why did we choose F# over C#?
- A high-level overview of the architecture of our push-based queryable API.
- Adopting F# for C#/OOP developers (inconveniences, C# interoperability, code style, DDD, TDD)

Publié dans : Formation
  • Login to see the comments

.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech

  1. 1. About me: @AntyaDev like types*
  2. 2. - Biggest provider in sport offering - Supports a lot of regulated markets - About 1000 microservices (200 distinct types) - 5 datacenters maintained fully by SBTech - About 500 concurrent live events at pick time - On average we handle about 100K+ RPS
  3. 3. highload and near real time
  4. 4. Why F#?
  5. 5. Why F#? If we have sexy C#
  6. 6. If we have sexy C# class Person { public string FirstName; // Not null public string? MiddleName; // May be null public string LastName; // Not null }
  7. 7. If we have sexy C# switch (shape) { case Circle c: WriteLine($"circle with radius {c.Radius}"); break; case Square s when (s.Length == s.Height): WriteLine($"{s.Length} x {s.Height} square"); break; case Rectangle r: WriteLine($"{r.Length} x {r.Height} rectangle"); break; }
  8. 8. If we have sexy C# public (string, string) LookupName(long id) { // some code.. return (first, middle); } var names = LookupName(id); WriteLine($"found {names.first} {names.last}.");
  9. 9. Why F#? If we have sexy C#
  10. 10. 1 day
  11. 11. Story of a new C# project Get("/api/bets", async request => { var result = Validator.Validate(request); // unit testable if (result.IsOk) { var response = await _httpClient.Call(request); return BetsMapper.MapResponse(response); // unit testable } else return Errors.MapError(result); // unit testable });
  12. 12. 1 day
  13. 13. In 1 week
  14. 14. public class BetsProviderService : IBetsProviderService { readonly IOpenBetsRepository _openBetsRepository; readonly ISettledBetsRepository _settledBetsRepository; public BetsProviderService(IOpenBetsRepository openBetsRepository, ISettledBetsRepository settledBetsRepository) { _openBetsRepository = openBetsRepository; _settledBetsRepository = settledBetsRepository; } } Story of a new C# project
  15. 15. Different Culture OOP FP abstraction extensibility purity composability correctness
  16. 16. Triggers: Much better tooling support Community growth Recursive modules
  17. 17. Tooling support Visual Studio RiderVisual Studio Code
  18. 18. Recursive modules type Plane() = member x.Start() = PlaneLogic.start(x) // types and modules can't currently be mutually // referential at all module private PlaneLogic = let start (x: Plane) = ...
  19. 19. Recursive modules module M type First = | T of Second and Second = | T of First
  20. 20. Recursive modules module M type First = | T of Second type Second = | T of First
  21. 21. Recursive modules module rec M type First = | T of Second type Second = | T of First
  22. 22. Recursive modules module M let private helper () = ... let publicApiFunc () = helper()
  23. 23. Recursive modules module rec M let publicApiFunc () = helper() let private helper () = ...
  24. 24. Community growth • To begin, F# has grown to be bigger than ever, at least as far as we can measure, through product telemetry, twitter activity, GitHub activity, and F# Software Foundation activity. • Active unique users of F# we can measure are in the tens of thousands. • Measured unique users of Visual Studio Code with Ionide increased by over 50% this year, to become far larger than ever. • Measured unique users of Visual Studio who use F# increased by over 20% since last year to be larger than ever, despite quality issues earlier in the year that we believe have inhibited growth. • Much of the measured growth coincides with the release of .NET Core 2.0, which has shown significant interest in the F# community.
  25. 25. namespace SportData.ETL.Core type ItemRawUpdate<'Item> = { Id: string Data: 'Item option } module TransformationFlow = let getUpdates (feed: ChangeFeed, lastOffset: uint32) = Code Style
  26. 26. Code sample let createGame (globalCache: GlobalItemsCache, masterEvent: MasterEventItem, currentTime: DateTime) = trial { let! league = Cache.get(globalCache.Leagues, masterEvent.LeagueID) let! country = Cache.get(globalCache.Countries, league.CountryID) let! sport = Cache.get(globalCache.Branches, masterEvent.BranchID) ... return { Id = masterEvent.ID.ToString() Type = getEventType(masterEvent) } }
  27. 27. I need website Website SBTech story
  28. 28. I need an API to build unique UI The Problem Oops?!
  29. 29. The Problem Oops?! 1. well defined contracts (Event, Market, Selection) 2. flexible way to query data and subscribe on changes 3. push updates notifications about changes 4. near real time change delivery (1 sec delay) We need to provide:
  30. 30. PUSH based Queryable API (Change Feed)
  31. 31. select * from Games where isLive = true order by totalBets desc limit 10 PUSH changes
  32. 32. Change Feed (Actor) Change Stream Feed View 2:2 1:0 Chelsea - Milan Milan - Liverpool
  33. 33. Subscribers Feed View 2:2 1:0 Chelsea - Milan Milan - Liverpool Change Log Query Change Feed (Actor)
  34. 34. Subscribers type NodeName = string type LastHeartBeat = DateTime type NodeSubscribers = Map<NodeName, LastHeartBeat> type FeedStatus = | ReadyForDeactivation | HasSubscribers | NoSubscribers type ChangeFeed = { Id: FeedId View: FeedView ChangeLog: ChangeLog Subscribers: NodeSubscribers Query: Query Status: FeedStatus } Query Feed View 2:2 1:0 Chelsea - Milan Milan - Liverpool Change Feed Change Log
  35. 35. module ChangeFeed let getSnapshot (feed: ChangeFeed, fastPreloadAmount: uint32) = ... let getUpdates (feed: ChangeFeed, lastOffset: uint32) = ... let getLatestUpdate (feed: ChangeFeed) = ... let subscribe (feed: ChangeFeed, name: NodeName, currentTime: DateTime) = ... let subscribeByOffset (feed: ChangeFeed, name: NodeName, lastOffset: uint32) = ... let unsubscribe (feed: ChangeFeed, name: NodeName) = ... let getInactiveNodes (feed: ChangeFeed, currentTime: DateTime, inactivePeriod: TimeSpan) = ... let reloadView (feed: ChangeFeed, queryResults: QueryResult seq) = ...
  36. 36. Feed View 2:2 1:0 Chelsea - Milan Milan - Liverpool type PartialView = { EntityType: EntityType Entities: IEntity seq } type FeedView = { OrderType: EntityType OrderIds: string seq OrderFilter: FilterFn option Views: Map<EntityType, PartialView> } let tryCreate (queryResults: QueryResult seq, orderType: EntityType, orderFilter: FilterFn option) = match queryResults with | NotMatchFor orderType -> fail <| Errors.OrderTypeIsNotMatch | Empty -> ok <| { OrderType = orderType; OrderIds = Seq.empty OrderFilter = orderFilter; Views = views } | _ -> ok <| ...
  37. 37. [<Property>] let ``empty ChangeLog should start with offset 0 and requested maxSize`` (maxSize: uint32) = let changeLog = ChangeLog.create(maxSize) Assert.Equal(0u, changeLog.Offset) Assert.Equal(maxSize, changeLog.MaxSize) [<Property(Arbitrary=[|typeof<Generators.Generator>|])>] let ``all changes in changeLog.stream should have UpdateType.Update`` (payloads: Payload array) = let mutable changeLog = ChangeLog.create(5u) for p in payloads do changeLog <- ChangeLog.append(changeLog, p) let result = changeLog.Stream.All(fun change -> change.Type = UpdateType.Update) Assert.True(result) TDD without Mocks
  38. 38. We have our own frontend but we need a Data The Next Challenge Oops?!
  39. 39. SBTech All changes Operator
  40. 40. RDBMS All changes from 40 tables ETL Operator Well defined entities compose entities react on changes recalculate odds (data locality)
  41. 41. C# F#Asp NET Core Orleans DB Drivers [ DDD; Tests] Infrastructure business logic
  42. 42. public Task<FullFeedUpdate> GetSnapshot(GetSnapshot msg) { // invoke F# code from C# var updates = ChangeFeedModule.getSnapshot(FeedState, msg.FastPreloadAmount); return updates; }
  43. 43. Applied at the heart of the system Zero Null Reference exceptions Domain errors instead of exceptions DDD with types (strong determinism) Dependency Rejection TDD without mocks (property based testing)
  44. 44. http://nbomber.com
  45. 45. - Technology agnostic (no dependency on HTTP, WebSockets, SSE) - Very simple API (to test Pull and Push scenarios) - CI/CD integration
  46. 46. - Technology agnostic (no dependency on HTTP, WebSockets, SSE) - Very simple API (to test Pull and Push scenarios) - CI/CD integration
  47. 47. - Technology agnostic (no dependency on HTTP, WebSockets, SSE) - Very simple API (to test Pull and Push scenarios) - CI/CD integration
  48. 48. - Technology agnostic (no dependency on HTTP, WebSockets, SSE) - Very simple API (to test Pull and Push scenarios) - CI/CD integration
  49. 49. type Step = | Request of RequestStep // to model Request-response pattern | Listener of ListenerStep // to model Pub/Sub pattern | Pause of TimeSpan // to model pause in your test flow type TestFlow = { FlowName: string Steps: Step[] ConcurrentCopies: int } type Scenario = { ScenarioName: string TestInit: Step option TestFlows: TestFlow[] Duration: TimeSpan }
  50. 50. let execStep (step: Step, req: Request, timer: Stopwatch) = task { timer.Restart() match step with | Request r -> let! resp = r.Execute(req) timer.Stop() let latency = Convert.ToInt64(timer.Elapsed.TotalMilliseconds) return (resp, latency) | Listener l -> let listener = l.Listeners.Get(req.CorrelationId) let! resp = listener.GetResponse() timer.Stop() let latency = Convert.ToInt64(timer.Elapsed.TotalMilliseconds) return (resp, latency) | Pause time -> do! Task.Delay(time) return (Response.Ok(req), 0L) }
  51. 51. type TestFlowRunner(flow: TestFlow) = let createActors (flow: TestFlow) = flow.CorrelationIds |> Set.toArray |> Array.map(fun id -> TestFlowActor(id, flow)) let actors = createActors(flow) member x.Run() = actors |> Array.iter(fun x -> x.Run()) member x.Stop() = actors |> Array.iter(fun x -> x.Stop()) member x.GetResult() = actors |> Array.collect(fun actor -> actor.GetResults()) |> TestFlowStats.create(flow)
  52. 52. // C# example var flow = new TestFlow(flowName: "READ Users", steps: new[] {step1, step2, step3}, concurrentCopies: 10); new ScenarioBuilder(scenarioName: "Test MongoDb") .AddTestFlow(flow) .Build(duration: TimeSpan.FromSeconds(10)) .RunInConsole();
  53. 53. // F# example let flow = { FlowName = "GET flow" Steps = [step1; step2; step3] ConcurrentCopies = 100 } Scenario.create("Test HTTP https://github.com") |> Scenario.addTestFlow(flow) |> Scenario.withDuration(TimeSpan.FromSeconds(10.0)) |> Scenario.runInConsole
  54. 54. var httpClient = new HttpClient(); var httpStep = Step.CreateRequest("GET request", execute: async _ => { var response = await httpClient.SendAsync(request); return response.IsSuccessStatusCode ? Response.Ok() : Response.Fail(); }); var pause_1s = Step.CreatePause(duration: TimeSpan.FromSeconds(1)); var scenario = new ScenarioBuilder("Test HTTP") .AddTestFlow("PULL", new[] { httpStep, pause_1s }, concurrentCopies: 10) .Build(duration: TimeSpan.FromSeconds(10)); scenario.RunInConsole();
  55. 55. RDBMS Storage Kafka ETL Kafka NoSQL Storage Push API
  56. 56. - 1470 LOC - (4) OOP objects - (~100) functions - try/catch used 2 times
  57. 57. async { let! x = myFunction() match x with | Some v -> ... | None -> ... } F# vNext match!
  58. 58. async { match! myFunction() with | Some v -> ... | None -> ... } F# vNext match!
  59. 59. F# vNext Struct Records type Vector3 = { X: float; Y: float; Z: float } [<Struct>] type Vector3 = { X: float; Y: float; Z: float }
  60. 60. F# vNext Struct Records [<Struct>] type Result<'T,'TError> = | Ok of 'T | Error of 'Terror [<Struct>] type ValueOption<'T> = | Some of ‘T | None
  61. 61. F# vNext Anonymous Records type Data = { X: int Y: string } let data = { X = 1; Y = "abc" }
  62. 62. F# vNext Anonymous Records let data = {| X = 1; Y = "abc" |}
  63. 63. F# vNext Structural Type System type User() = member this.GetName() = "test user" type Car() = member this.GetName() = "test car" let inline printName (x: ^T when ^T : (member GetName: unit -> string)) = x.GetName() |> Console.WriteLine
  64. 64. F# vNext Applicative functors validate { let! name = validateName(name) let! age = validateAge(age) let! email = validateEmail(email) return User(name, age, email) }
  65. 65. F# vNext Applicative functors validate { let! name = validateName(name) and! age = validateAge(age) and! email = validateEmail(email) return User(name, age, email) }
  66. 66. F# vNext Applicative functors parallel { let! x = slowRequestX() and! y = slowRequestY() and! z = slowRequestZ() return f(x, y, z) }
  67. 67. THANKS @antyadev antyadev@gmail.com @nbombertest http://nbomber.com

×