Behind the Curtain

Testing This

TESTING TESTING

TESTING TESTING

Thursday, April 21, 2011

ZOMG!


GOOD MORNING!


Aaron Patterson


@tenderlove


WWFMD?

ITWFMWD?

AT&T, AT&T logo and all AT&T related marks are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies.

Miami Vice Principal
Señor Facebook
Integration
Engineering Manager


Elect


Our Time Together

12% 2%
24%

61%

Inspirational Entertaining Informative Wasted


KEYNOTE


Our Time Together
10%
20%

20%
50%

Inspirational Entertaining Informative Wasted


20% More
Transitions


I enjoy dealing
with behinds
the scenes
information

Living Behind
the Curtain 6 subjects, less
than 10
minutes on each
of them


DATABASE

APPLICATION

CLIENT

Code as Data


Politics


We don't vote on
commits. It
would be too
inefficient.

"Open Source
is a Democracy"


OSS is an Oligarchy


Benevolent Dictator


Small projects Large projects have
have few people, too many people so
so it is efficient it becomes
inefficient to
individually consider
all feedback

Tyrannical Leader


Care and Feeding of
Your Tyrant


Cheetos


Good
reproduction
steps

Feed them Quality
Bugs


Feed them Patches
with Tests


Gemspec,
Jewler
Hoe,
etc

leave build systems


leave test
frameworks


Use +1's Carefully


320 comments


210 plus one


FFFFFFFUUUUU
UUUUUUU


Superﬁcial


Merit


Quality


We'll discuss
this in closing.

How to ﬁx?


Encodings
the silent killer


Data goes in, Data goes out

Never a miscommunication

Until there is


`encode': "xE9" from ASCII-8BIT to UTF-8
(Encoding::UndefinedConversionError)


Encoding Error?

You can't explain that

String Tagging


Strings with
unknown
encoding are
marked as binary.

"hello! <3".encoding # => #<Encoding:UTF-8>


socket.read(4).encoding
# => #<Encoding:ASCII-8BIT>


To understand
these bugs, we
must find
boundaries

Boundaries


App Code


Sadness
Web Server
Belt

NoSQL App Code File System

Database


Databases Post contains
know the encoding /
stored encoding spec declares
default

#<Encoding:ASCII-8BIT>
is usually a bug!


# encoding: utf-8

name = ' '
user = User.create!(:name => name)
user.reload
user.name + name


# encoding: utf-8

name = ' '
user.reload
user.name + name

incompatible character encodings: ASCII-8BIT
and UTF-8 (Encoding::CompatibilityError)


>> name = "Aaron Patterson"
>> name.encoding
=> #<Encoding:UTF-8>
>> User.create!(:name => name)
>> u = User.find :first
>> u.name.encoding
=> #<Encoding:ASCII-8BIT>


Ruby attempts
to convert the
binary for you

name = 'aaron patterson'
user.reload
user.name + name


Ruby attempts
to convert the
binary for you

name = 'aaron patterson'
user.reload
user.name + name

no error


ASCII-8BIT + other
should raise


Aaro
Opin n's
ion
Cor
ner!

ASCII-8BIT + other
should raise


Resources Know
Encoding


Prepared Statements


SELECT * FROM "users" WHERE id = 1


Normal Queries

Parse the Query
Formulate an Execution Plan
Return results


Application

Database


Application

Result Set

Database


Application

query: 1234

Database


Preparation

Parse the Query
Formulate an Execution Plan


Execution

Return Results


Data transfer
decreases


Query Planner
Improves


Security


Talk about
problem
inserting rows
with session
data

SQLite3 Issues :'(
Could insert
session data,
but could not
find it


INSERT INTO "sessions"
("data", "session_id")
VALUES (?, ?)

['data',
'BAh7BjoIZm9vSSIIYmF6BjoGRUY=']
['session_id',
'dae0fb8a9c34e6980c9d0fae33a8fff6']


db = SQLite3::Database.new ':memory:'
db.trace { |sql| puts sql }


("data", "session_id")

VALUES (

'BAh7BjoIZm9vSSIIYmF6BjoGRUY=',

x'3633303337623066376536613130343132343
765623763626631616439303631')


Made a sample
program to
reproduce the
db.trace { |sql| puts sql } problem

stmt = db.prepare(
'INSERT INTO "sessions"
(session_id)
VALUES (?)')

stmt.bind_param 1, 'fuu'
stmt.execute


Resulting
Query

(session_id) VALUES ('fuu')


stmt.bind_param 1, 'fuu'.encode('BINARY')
stmt.execute


INSERT INTO "sessions" (session_id)
VALUES (x'667575')


Why did sqlite3 do
this?

Talk about column
affinity

INSERT INTO "sessions" (session_id)
VALUES (x'667575')


Why is it stored as
Binary?


Why is the session id
tagged binary?


>> x = OpenSSL::Random.random_bytes(10)
=> "Gx93xFCxB2xCExC0xECxBBxA7W"
>> x.encoding


>> y = x.unpack('H*')
=> ["4793fcb2cec0ecbba757"]
>> y.first.encoding


def generate_sid
ActiveSupport::SecureRandom.hex(16)
end


Should hex have tagged
it?

Should have generate_sid
tagged it?

def generate_sid
sid = ActiveSupport::SecureRandom.hex(16)
if sid.respond_to?(:encode!)
sid.encode!('UTF-8')
end
sid
end


Find the Source


ARel and
ActiveRecord


Fear of SQL


Why?


But not
specifically SQL

Working with
sets

We must learn SQL
Can fetch
correct data.


We must learn to
work with Sets


Avoid SQL for:


code reuse


security


from databases

independence


What is ARel?


Represents SQL
as a Tree

AST Represents the
IDEA of a SQL
statement


SQL Compiler


Only until we
invoke the
compiler, does the
AST become a
query

Represents
an IDEA


Relationship to
ActiveRecord


User.where('something')


class User
scope :heart where(:name => '<3')
end

User.heart.select('name')


Our Application

ActiveRecord

ARel Database


Our Application

.to_a
ActiveRecord

ARel Database


Our Application

ActiveRecord

to_sql
ARel Database


Our Application

ActiveRecord

SELECT * ...

ARel Database


Our Application

ActiveRecord

[{: name => '<3'}]

ARel Database


ActiveRecord::Relation


shortens to...


ARel


ARel != ARel


:'(


Future we can
have
optimizers,
compiler cache,
etc

SQL Compiler


Interpreter?


module Arel
module Visitors
class Mongo < Arel::Visitors::Visitor
attr_reader :db

def initialize db; @db = db; end
Query = Struct.new(:collection_name, :fields, :conditions)

private

def visit_Arel_Nodes_SelectStatement o
o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query|
collection = db.collection query.collection_name
fields = query.fields
selector = Hash[query.conditions]
opts = {}
opts[:fields] = fields unless fields.empty?
opts[:limit] = o.limit.expr.to_i if o.limit
collection.find(selector, opts).to_a
}.flatten
end

def visit_Arel_Nodes_SelectCore o
fields = o.projections.map { |proj| visit(proj) }.compact
conditions = o.wheres.map { |condition| visit(condition) }.flatten(1)
Query.new(visit(o.source), fields, conditions)
end

def visit_Arel_Nodes_And o
o.children.map { |child| visit child }
end

def visit_Arel_Nodes_Equality o
[visit(o.left), visit(o.right)]
end

def visit_Arel_Attributes_Attribute o
return if o.name == '*'; o.name
end

def visit_Arel_Nodes_JoinSource o
visit o.left
end

def visit_Arel_Table o
o.name
end
def literal o; o; end
alias :visit_String :literal
alias :visit_Fixnum :literal
end
end
end


module Arel
module Visitors
class Mongo < Arel::Visitors::Visitor
attr_reader :db

def initialize db; @db = db; end
Query = Struct.new(:collection_name, :fields, :conditions)

private

def visit_Arel_Nodes_SelectStatement o
o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query|
collection = db.collection query.collection_name
fields = query.fields
selector = Hash[query.conditions]
opts = {}
opts[:fields] = fields unless fields.empty?
opts[:limit] = o.limit.expr.to_i if o.limit
collection.find(selector, opts).to_a
}.flatten
end

Mongo Interpreter
def visit_Arel_Nodes_SelectCore o
fields = o.projections.map { |proj| visit(proj) }.compact
conditions = o.wheres.map { |condition| visit(condition) }.flatten(1)
Query.new(visit(o.source), fields, conditions)
end

64 loc
def visit_Arel_Nodes_And o
o.children.map { |child| visit child }
end

def visit_Arel_Nodes_Equality o
[visit(o.left), visit(o.right)]
end

def visit_Arel_Attributes_Attribute o
return if o.name == '*'; o.name
end

def visit_Arel_Nodes_JoinSource o
visit o.left
end

def visit_Arel_Table o
o.name
end
def literal o; o; end
alias :visit_String :literal
alias :visit_Fixnum :literal
end
end
end


gist.github.com/845782


Custom DSL


Responds to *

class Select < Struct.new(:columns)
def self.* other
other.select = new(Arel.sql('*'))
other
end
end


Responds to WHERE

class From < Struct.new(:table, :conditions)
def WHERE conditions
self.conditions = conditions
Where.new(self)
end
end


Responds to to_s

class Where < Struct.new(:from, :select)
def to_s
Arel::Table.engine = Arel::Sql::Engine.new(
FakeRecord::Base.new)
table = Arel::Table.new from.table
table.project(select.columns).where(
from.conditions.map { |k,v| table[k].eq v }).to_sql
end
end


Bootstrap Methods

SELECT = Select
def FROM table
From.new table
end


x = SELECT * FROM("users") .WHERE(:id => 10)
puts x


Write SQL Ruby
to avoid SQL


Talked about
data from the
back end, talk a
bit about data to
the front end

Streaming


Twitter Streaming API


Needed for
persistent
connections

Chunked Responses


Normal Timeline
Buffers data,
then sends a
response


Normal Timeline
Buffers data,
then sends a
response

Processing ERb


Normal Timeline
Buffers data,
then sends a
response

Processing ERb

Download Asset

Download Asset


Chunked Timeline
Send data as
soon as it is
available


Chunked Timeline
Send data as
soon as it is
available
Processing ERb


Chunked Timeline
Send data as
soon as it is
available
Processing ERb

Download Asset

Download Asset


Implementation


Rack API


status, headers, body = app.call(env)


status.to_i


headers.is_a?(Hash)


body.responds_to?(:each)


Inside Rack
Calls each,

possibly calls
close
body.each do |chunk|
output(chunk)
end
body.close rescue nil


Sample Application
class MyApp
def call(env)
# some computation
body = 'hello'
# more computation
body << ' world'

[200, { 'X-Hello' => 'World' }, [body]]
end
end


Delay work until each
class FooBody
def each
yield "hello "
sleep(10) # simulate work
yield "world!"
end
end


Database
connections
were lost

Problems ☹


Database
connections are
managed in
middleware

Middleware


"There are two hard problems
in CS: cache invalidation
and naming things"
-- Phil Karlton


class DbCache < Struct.new(:app)
def call(env)
# init cache
status, headers, body = app.call(env)
# clear cache
[status, headers, body]
end
end


class BodyProxy < Struct.new(:delegate)
def each
delegate.each { |x| yield x }
end

def close
delegate.close rescue nil
# clear cache
end
end


Update Middleware,
Database Works! ☺


Same problem in
query cache as
the database
handle cache.

But Query Cache
was broken. ☹


Middleware is
coupled to the
callstack


~ 24 middleware


~ 5 required a proxy


Types of Middleware

Generators Rack jams these to
one API.

Filters Approach seems
naive when
examining usage
Lifecycle Handlers
Tied to
callstack


Solutions!


Embrace The
Differences!


Lifecycle Listeners

class Listener
def created(event)
end

def destroyed(event)
end
end


Generators Synchronous or
Asynchronous

class Resource
def service(request, response)
10.times { |i|
response.body.write "hello #{i}"
}
response.body.close
Return value is
end ignored
end


Filters

class Filter
def filter(request, response, chain)
chain.filter(request, response)
end
end


Synchronous

Asynchronous

Freedom from the Callstack

Thread Safety


I stole this API.


TEE HEE!!!!!


ﬁxing our tyrants


Facebook + Github


Get Involved


Only fix the
immediate
problem

Gain Trust


Then Persuade


Questions?


Behind the Curtain

Recommandé

Recommandé

Contenu connexe

Similaire à Behind the Curtain

Similaire à Behind the Curtain (20)

Plus de Aaron Patterson

Plus de Aaron Patterson (9)

Dernier

Dernier (20)

Behind the Curtain