Parse search forms from applications and generate SQL

1. 23.08.14 - Page 1
Département
Office
SQL::Abstract::FromQuery
Building DB requests from Web queries
YAPC::EU::2014, Sofia
laurent.dami@justice.ge.ch
Etat de Genève, Pouvoir Judiciaire Département
Office

2. Power search : the "widget" way
Source: https://rt.cpan.org/Search/Build.html

3. Power search : the "parsed" way
Name : Smi*
Gender : M
Salary : > 4000
Job : ! programmer, analyst
Birth : BETWEEN 01.01.1970 AND 31.12.1990
Address: ! NULL

4. Agenda
• Background : SQL::Abstract & SQL::Abstract::More
• SQL::Abstract::FromQuery
– purpose
– API
• internals
– Regexp::Grammars
– multiple inheritance

5. Background : SQL::Abstract
SQL::Abstract::More
Translating Perl datastructures
into SQL

6. SQL::Abstract ('SQLA') : example
my $sqla = SQL::Abstract->new;
my ($sql, @bind) = $sqla->select(
$table_name,
[qw/col1 col2 col3/],
{ col1 => 'val1',
col2 => {-in => [qw/foo bar/]},
col3 => {-between => [qw/2012-01-01 2012-07-01/]},
col4 => {">" => 999},
col5 => {-like => 'foo%'},
},
[qw/col1 col2/],
);
# SELECT col1, col2, col3 FROM $table_name
# WHERE col1 = ? AND col2 IN (?, ?) AND col3 … etc
# ORDER BY col1, col2

7. Purpose
• generate SQL from Perl data structures
– direct from config or from user input
– avoid painful string manipulations with regex / join / etc.
• placeholders and bind values
– avoid datatype distinctions (strings / numbers /dates / etc.)
– avoid SQL injection
  used by DBIx::Class, DBIx::DataModel, …

8. "Special operators" in SQLA
• support for user-defined SQL extensions. For ex :
– fulltext index
– multivalued columns
– case-insensitive
– LIKE / GLOB / etc.
• useful for DBMS-independent queries in apps
– the "special operator" translates to DBMS-specific syntax
{ col1 => {-contains => [qw/YAPC Sofia/]},
col2 => {-anyval => [12, 34, 56]},
col3 => {-glob => 'foo*'},
}

9. SQL::Abstract::More ('SQLAM')
• fully compatible with SQLA
• named parameters
• more SQL constructs (group by, limit/offset, union/intersect, etc.)
• more syntax (aliases, joins, ordering direction, etc.)
my $sqla = SQL::Abstract::More->new;
my ($sql, @bind) = $sqla->select(
-columns => [qw/table1.col|ct1 table2.col|ct2 … /],
-from => $table_name,
-where => { col1 => 'val1',
col2 => {-in => [qw/foo bar/]},
… },
-union => [-columns => …, -from => …, -where => …],
-order_by => [qw/+col1 -col2/],
);

10. 23.08.14 - Page 1
Département
Office
SQLA, SQLAM are for programmers
 what about users ?
SQL
Perl
datastructure
SQLA, SQLAMHTTP
query
SQLA::FromQuery

11. HTTP query
Name : Smi*
Gender : M
Salary : > 4000
Job : ! programmer, analyst
Birth : BETWEEN 01.01.1970 AND 31.12.1990
Address: ! NULL

12. Expected SQLA structure
{
Name => {-like => 'Smi%'},
Gender => 'M',
Salary => {'>' => 4000},
Job => {-not_in => [qw/programmer analyst/]},
Birth => {-between => [qw/1970-01-01 1990-12-31/]},
Address => {'<>' => undef},
}
SELECT * FROM people
WHERE Name LIKE 'Smi%'
AND Gender = 'M'
AND Salary > 4000
AND Job NOT IN ('programmer', 'analyst')
AND Birth BETWEEN 1970-01-01 AND 1990-12-31
AND Address IS NOT NULL

13. SYNOPSIS
my $parser = SQL::Abstract::FromQuery->new();
my $criteria = $parser->parse($http_query);
# $http_query is an object with a ->param() method
# .. or it can be just a hashref
my ($sql, @bind)
= $sqla->select($table, @cols, $criteria);
# or
my ($sql, @bind) = $sqlam->select(
-columns => @cols,
-from => $table,
-where => $criteria,
);

14. Options
my $parser = SQL::Abstract::FromQuery->new(
# additional components (optional)
-components => [qw/FR Oracle/],
# grammar rules for some specific fields (optional)
-fields => {
standard => [qw/field1 field2 .../],
bool => [qw/bool_field1/],
...
}
);

15. Components
• FR
– dates au format dd.mm.yyyy
– mots-clés en français (VRAI, FAUX, ENTRE, NUL, etc.)
– messages d'erreur en français
• Oracle
– queries on date and time
• ["to_date(?, '$date_format')", $date]
• ["to_date(?, '$time_format')", $date]
 WHERE D_BIRTH > TO_DATE('DD.MM.YYYY', ?)
• Contains
– generate fulltext queries
• … (user-defined)

16. Field types : which grammar to apply
• Standard
– a plain value (number, string, date or time).
– a list of values, separated by ','.
– a negated value or list of values; negation is expressed by ! or != or - or
<>
– a comparison operator <=, <, >=, > followed by a plain value
– the special word NULL
– BETWEEN val1 AND val2
– boolean values YES, NO, TRUE or FALSE
• Restricted
– string / bool / date / … (user-defined)

17. Internals

18. Regexp::Grammars
use Regexp::Grammars; qr{
<grammar: SQL::Abstract::FromQuery>
<rule: standard>
<MATCH=between> | <MATCH=op_and_value> | <MATCH=values>
<rule: values>
<[value]>+ % ,
<token: compare>
<= | < | >= | >
<rule: value>
<MATCH=null> | <MATCH=date> | <MATCH=time> | <MATCH=string>
<rule: date>
<day=(dd?)>.<month=(dd?)>.<year=(ddd?d?)>
| <year=(ddd?d?)>-<month=(dd?)>-<day=(dd?)>

19. Autoactions hooked to the grammar
sub between {
my ($self, $h) = @_;
return {-between => [$h->{min}, $h->{max}]};
}
sub values {
my ($self, $h) = @_;
my $n_values = @{$h->{value}};
return $n_values > 1 ? {-in => $h->{value}}
: $h->{value}[0];
}

20. Grammar Inheritance
<grammar: SQL::Abstract::FromQuery::FR>
<extends: SQL::Abstract::FromQuery>
<rule: null>
NULL?
<rule: between>
<SQL::Abstract::FromQuery::between>
| ENTRE (*COMMIT) (?: <min=value> ET <max=value> | <error:> )
<rule: bool>
O(?:UI)? (?{ $MATCH = 1 })
| V(?:RAI)? (?{ $MATCH = 1 })
| NO?N? (?{ $MATCH = 0 })
| F(?:AUX|ALSE)? (?{ $MATCH = 0 })
| Y(?:ES)? (?{ $MATCH = 1 })

21. Rules inheritance
package SQL::Abstract::FromQuery::Oracle;
use parent 'SQL::Abstract::FromQuery';
use mro 'c3';
¨sub date {
my ($self, $h) = @_;
my $date_format = $self->{date_format}
|| 'YYYY-MM-DD';
my $date = $self->next::method($h);
return ["to_date(?, '$date_format')", $date];
}

22. 23.08.14 - Page 1
Département
Office
Conclusion

23. Features
• flexible syntax for user input
– plain value
– list of values
– comparison operators
– patterns
– special constructs (BETWEEN, MATCH, etc.)
– Including SQLA "special ops"
• internationalization
• automate specific behaviours for some form fields
– data conversion
– inject special ops

24. Usage
• See App::AutoCRUD
  come to tomorrow's talk 