1. What LINQ is aboutor, Playingwith LINQ, MongoDB, and someold new patterns for developers.or, How to injectfunctionalparadigms in imperativelanguagesor, parallelism made easy. WebWorkersCamp, 03/07/2010 Pierre Couzy – Microsoft France
2. Who are you ? Worked in the Microsoft space for 20 years Speaker, trainer, developer, architect, … Primarilyinterested in Web topics Identity, architecture, WS-*, interop, cloud, … Recentinterop/OSS activity Worked on drupal for Sql server project Works on automateddrupaltesting on Windows Speaker at major PHP events DrupalCon, PHP Forum, Symfony live blog.couzy.com / pierre.couzy@microsoft.com
3. Agenda LINQ More LINQ LINQ for MongoDB GoingParallel GoingDistributed Q&A Disclosure: This session is heavily Microsoft-Biased (which is to be expected, I work there)
4. LINQ : Why ? C# is an imperativelanguage Whichisnice Except for a few things Splitting data manipulation betweenyour code and your data store Coding data manipulation Applying new manipulation patterns to existing data
6. Whatwe’dlikeis Express more intent, lessimplementation Easy composition Strongtyping This issubject to hot debate Be as declarative as possible within the limitsimposed by C#
8. Things to notice Lazyevaluation If westepintothis code, wesee LINQ triggers itsmagic on demand Easy composition If weadd new clauses after the LINQ expression isbuilt but beforeit’sconsumed, wesimplymerge the additions to the existing expression So far, we’veworked on objects Whichis how manynaive LINQ implementationswork : synchronously pipelining methods calls on collections LINQ has an interesting twist Expression trees
9. Expression Trees and IQueryable Let’stake the sameexample and introduceAsQueryable() Uponexecution, wecan notice the compiler didn’teatour LINQ expression It’sstill in there, using a tree format (an AbstractSymbolTree) That treewillberesolvedatruntime In the Linq to Objects case, wegetexactly the samebehaviour. The iteration pattern maystillbeyield-based, but thisisleft to IQueryable’simplementationdetails.
10. Playingwith expression trees Let’stackleLinq To SQL My expression treeisnowpushed to the database layer If I want to block the flow to SQL, I just have to insert a new node in the tree. Whichallows me to balance things Using C# patterns and methodswhen I wantthem Pushing to the databasewhatitdoes best
11. IQueryable goodies So, if I wantmythings to be LINQ aware, I caneither Buildsomemethods and let the IEnumerable<T> do its pure objectmagic (I stillgetlazyeval) Or, playwithIQueryable And do myowninterpretation of the tree Let’sseeMongoDB in LINQ
12. Strongly-typed interaction when querying and updating collections. Improved interface to send common Mongo commands (creating indices, getting all the existing dbs and collections, etc.). Ultra-fast de/serialization of BSON to .Net CLR types and back. LINQ-to-Mongo Support for Mono
13. LINQ Patterns Having a LINQ Expression isnice: youbasicallydelegate the execution to someoneelse’s code Plus, your LINQ Expressions have every good aspect youusuallyexpectfromfunctionalprogramming (ie, they are closures). So, they are perfectlysuited to parallelisation All you have to do isaddanothernode to the expression tree Demo : AsParallel()
14. WritingMapReduce in LINQ public static IQueryable<R> MapReduce<S,M,K,R>(this IQueryable<S> source, Expression<Func<S,IEnumerable<M>>> mapper, Expression<Func<M,K>> keySelector, Expression<Func<K,IEnumerable<M>,R>> reducer){ return source.SelectMany(mapper).GroupBy(keySelector, reducer);} Notice that the callingsyntax : Source.MapReduce(lMapper, lKeySelector, lReducer)issimply a new nodethatcanbeinserted in a larger expression tree. This MapReducemethodcanrun on N coresusing PLINQ, or on X machines (and N cores per machine) usingDryadLINQ
15. Collection<T> collection; boolIsLegal(Key k); string Hash(Key); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value}; DryadLINQ = LINQ + Dryad Vertexcode Queryplan (Dryad job) Data collection C# C# C# C# results
16. Parallel execution var logentries = from line in logs where !line.StartsWith("#") select new LogEntry(line); var user = from access in logentries where access.user.EndsWith(@"lfar") select access; var accesses = from access in user group access by access.page into pages select new UserPageCount("ulfar", pages.Key, pages.Count()); var htmAccesses = from access in accesses where access.page.EndsWith(".htm") orderby access.count descending select access;
