12.04.23 - Page 1

DépartementOffice

DBIx::Classvs.

DBIx::DataModel

FPW 2012, Strasbourg

laurent.dami@justice.ge.chDépartement

Office

Agenda

• Introduction• Schema definition• Data retrieval• Joins• Classes and methods• Resultset/Statement objects• Customization• Conclusion

Introduction

ORM layer

Database

DBD driver

DBI

Object-Relational Mapper

Perl program

ORM useful for …

• dynamic SQL generation– navigation between tables– generate complex SQL queries from Perl datastructures– better than phrasebook or string concatenation

• automatic data conversions (inflation / deflation)• transaction encapsulation • data validation• computed fields• caching• expansion of tree data structures coded in the relational

model• …

CPAN ORM Landscape

• Discussed here– DBIx::Class (a.k.a. DBIC)– DBIx::DataModel (a.k.a. DBIDM)

• Many other– Rose::DB, Jifty::DBI, Fey::ORM, ORM,

DBIx::ORM::Declarative, Tangram, Coat::Persistent, ORLite,DBR, DBIx::Sunny, DBIx::Skinny, DBI::Easy, …

DISCLAIMER- I'm not an expert of DBIC- I'll try to be neutral in comparisons, but …- Won't cover all topics

A bit of 2005 historyClass::DBI (1999)

Class::DBI::Sweet (29.05.05)- SQL::Abstract

DBIx::DataModel 0.10 (16.09.05)

SQL::Abstract (2001)

DBIx::Class 0.01 (08.08.05)- SQL::Abstract

DBIx::Class 0.03 (19.09.05)- prefetch joins

Class::DBI::Sweet 0.02 (29.06.05)-prefetch joins

DBIx::DataModel private (feb.05)

Some figures

• DBIC– thousands of users– dozens of contributors– 162 files– 167 packages– 1042 subs/methods– 16759 lines of Perl– 1817 comment lines– 19589 lines of POD– 70 transitive dependencies

• DBIDM– a couple of users– 1 contributor– 31 files– 40 packages– 175 subs/methods– 3098 lines of Perl– 613 comment lines– 8146 lines of POD– 54 transitive dependencies

straight from CPAN::SQLite

Our example : CPAN model

Author

Distribution Module

1

*

1 1..*

* *

contains ►

depends_on ►

12.04.23 - Page 1

DépartementOffice

Schema definition

12.04.23 - Page 1

DépartementOffice

DBIC Schema class

use utf8;package FPW12::DBIC;use strict;use warnings;

use base 'DBIx::Class::Schema';

__PACKAGE__->load_namespaces;

1;

12.04.23 - Page 1

DépartementOffice

DBIC 'Dist' table class

use utf8;package FPW12::DBIC::Result::Dist;use strict; use warnings;use base 'DBIx::Class::Core';

__PACKAGE__->table("dists");__PACKAGE__->add_columns(

dist_id => { data_type => "integer", is_nullable => 0, is_auto_increment => 1 },

dist_vers => { data_type => "varchar", is_nullable => 1, size => 20 }, ... # other columns );__PACKAGE__->set_primary_key("dist_id");__PACKAGE__->belongs_to ('auth', 'FPW12::DBIC::Result::Auth', 'auth_id');__PACKAGE__->has_many ('mods', 'FPW12::DBIC::Result::Mod', 'dist_id');__PACKAGE__->has_many ('depends', 'FPW12::DBIC::Result::Depends', 'dist_id');__PACKAGE__->many_to_many(prereq_mods => 'depends', 'mod');

# idem for classes Auth, Mod, etc.

DBIDM centralized declarations

use DBIx::DataModel;DBIx::DataModel->Schema("FPW12::DBIDM")# class table primary key # ===== ===== ===========->Table(qw/Auth auths auth_id /)->Table(qw/Dist dists dist_id /)->Table(qw/Mod mods mod_id /)->Table(qw/Depends depends dist_id mod_id/)# class role multipl. (optional join keys) # ===== ==== ======= ====================->Association([qw/Auth auth 1 auth_id /], [qw/Dist dists * auth_id /])->Composition([qw/Dist dist 1 /], [qw/Mod mods * /])->Association([qw/Dist dist 1 /], [qw/Depends depends * /])->Association([qw/Mod mod 1 /], [qw/Depends depends * /])->Association([qw/Dist used_in_dist * depends distrib /], [qw/Mod prereq_mods * depends mod /]);

12.04.23 - Page 1

DépartementOffice

Schema def. comparison

• DBIC– one file for each class– regular Perl classes– full column info– 1-way "relationships"

• belongs_to• has_many• may_have• ….

– custom join conditions• any operator• 'foreign' and 'self'

• DBIDM– centralized file– dynamic class creation – no column info (except pkey)– 2-ways "associations" with

• multiplicities• role names• diff association/composition

– custom column names• only equalities

12.04.23 - Page 1

DépartementOffice

Data retrieval

12.04.23 - Page 1

DépartementOffice

DBIC: Search

use FPW12::DBIC;

my $schema = FPW12::DBIC->connect($data_source);

my @dists = $schema->resultset('Dist')->search( {dist_name => {-like => 'DBIx%'}}, {columns => [qw/dist_name dist_vers/]},);

foreach my $dist (@dists) { printf "%s (%s)\n", $dist->dist_name, $dist->dist_vers;}

12.04.23 - Page 1

DépartementOffice

DBIDM: Search

use FPW12::DBIDM;use DBI;

my $datasource = "dbi:SQLite:dbname=../cpandb.sql";my $dbh = DBI->connect($datasource, "", "",

{RaiseError => 1, AutoCommit => 1});FPW12::DBIDM->dbh($dbh);

my $dists = FPW12::DBIDM::Dist->select( -columns => [qw/dist_name dist_vers/], -where => {dist_name => {-like => 'DBIx%'}},);

foreach my $dist (@$dists) { print "$dist->{dist_name} ($dist->{dist_vers})\n";}

12.04.23 - Page 1

DépartementOffice

Simple search comparison

• DBIC– schema is an object– result is a list or a resultset

object– accessor methods for

columns– uses SQL::Abstract

• mostly hidden

• DBIDM– schema is a class (default) or an

object– result is an arrayref (default)– hash entries for columns– uses SQL::Abstract::More

• user can supply a custom $sqla obj

12.04.23 - Page 1

DépartementOffice

SQL::Abstract new()

• special operators

– ex: DBMS-independent fulltext search

# where {field => {-contains => [qw/foo bar/]}

my $sqla = SQL::Abstract->new(special_ops => [ {regex => qr/^contains(:?_all|_any)?$/i, handler => sub { my ($self, $field, $op, $arg) = @_; my $connector = ($op =~ /any$/) ? ' | ' : ' & '; my @vals = ref $arg ? @$arg : ($arg); @vals = map { split /\s+/ } grep {$_} @vals; my $sql = sprintf "CONTAINS($field, '%s') > 0",

join $connector, @vals; return ($sql); # no @bind

}]);

12.04.23 - Page 1

DépartementOffice

DBIC: Find single record

my $auth1 = $schema->resultset('Auth')->find(123);

my $auth2 = $schema->resultset('Auth') ->find({cpanid => 'DAMI'});

12.04.23 - Page 1

DépartementOffice

DBIDM: Find single record

my $auth1 = FPW12::DBIDM::Auth->fetch(123);

my $auth2 = FPW12::DBIDM::Auth->search( -where => {cpanid => 'DAMI'}, -result_as => 'firstrow',);

12.04.23 - Page 1

DépartementOffice

DBIC: Search args

• 1st arg : "where" criteria• 2nd arg : attributes

– distinct– order_by– group_by, having– columns # list of columns to retrieve from main table– +columns # additional columns from joined tables or

from functions – join # see later– prefetch # see later– for– page, rows, offset, etc.– cache

DBIDM: Select args

my $result = $source->select( -columns => \@columns,, -where => \%where, -group_by => \@groupings, -having => \%criteria, -order_by => \@order, -for => 'read only', -post_SQL => sub { … }, -pre_exec => sub { … }, -post_exec => sub { … }, -post_bless => sub { … }, -page_size => …, -page_index => …, -limit => …, -offset => …, -column_types => \%types, -result_as => $result_type,);

DBIDM: Polymorphic result

-result_as =>– 'rows' (default) : arrayref of row objects– 'firstrow' : a single row object (or undef)– 'hashref' : hashref keyed by primary keys– [hashref => @cols] : cascaded hashref– 'flat_arrayref' : flattened values from each row– 'statement' : a statement object (iterator)– 'fast_statement' : statement reusing same memory– 'sth' : DBI statement handle– 'sql' : ($sql, @bind_values)– 'subquery' : \["($sql)", @bind]

don't need method variants : select_hashref(), select_arrayref(), etc.

DBIDM: Fast statement

• like a regular statement– but reuses the same memory location for each row– see DBI::bind_col()

my $statement = $source->select( . . . , -result_as => 'fast_statement');

while (my $row = $statement->next) { . . . # DO THIS : print $row->{col1}, $row->{col2} # BUT DON'T DO THIS : push @results, $row;}

Advanced search comparison

• DBIC– 'find' by any unique

constraint– "inner" vs "outer" columns– result is context-dependent– optional caching

• DBIDM– 'fetch' by primary key only– all columns equal citizens– polymorphic result– no caching– statement-specific inflators– callbacks

Joins

Task : list distributions and their authors

Author

Distribution Module

1

*

1 1..*

* *

contains ►

depends_on ►

DBIC: Join

my @dists = $schema->resultset('Dist')->search( {dist_name => {-like => 'DBIx%'}}, {columns => [qw/dist_name dist_vers/], join => 'auth',

'+columns' => [{'fullname' => 'auth.fullname'}], },);

foreach my $dist (@dists) { printf "%s (%s) by %s\n",

$dist->dist_name, $dist->dist_vers, $dist->get_column('fullname'); # no accessor meth}

DBIC: Join with prefetch

my @dists = $schema->resultset('Dist')->search( {dist_name => {-like => 'DBIx%'}}, {columns => [qw/dist_name dist_vers/], prefetch => 'auth', },);

foreach my $dist (@dists) { printf "%s (%s) by %s\n",

$dist->dist_name, $dist->dist_vers, $dist->auth->fullname; # already in memory}

DBIDM: Join

my $rows = FPW12::DBIDM->join(qw/Dist auth/)->select( -columns => [qw/dist_name dist_vers fullname/], -where => {dist_name => {-like => 'DBIx%'}},);

foreach my $row (@$rows) { print "$row->{dist_name} ($row->{dist_vers}) " . "by $row->{fullname}\n";}

Dist Auth

..::AutoJoin::…

DBIDM::Source::Join

new class created on the fly

Multiple joins

• DBIC

join => [ { abc => { def => 'ghi' } }, { jkl => 'mno' },

'pqr' ]

• DBIDM

->join(qw/Root abc def ghi jkl mno pqr/)

Join from an existing record

• DBIC

my $rs = $auth->dists(undef, {join|prefetch => 'mods'});

• DBIDM

my $rows = $auth->join(qw/dists mods/)->select;

DBIC: left/inner joins

• attribute when declaring relationships# in a Book class (where Author has_many Books)

__PACKAGE__->belongs_to( author => 'My::DBIC::Schema::Author', 'author', { join_type => 'left' } );

• cannot change later (when invoking the relationship)

DBIDM: Left / inner joins

->Association([qw/Author author 1 /], [qw/Distribution distribs 0..* /])

# default : LEFT OUTER JOIN

->Composition([qw/Distribution distrib 1 /], [qw/Module modules 1..* /]);

# default : INNER JOIN

# but defaults can be overriddenMy::DB->join([qw/Author <=> distribs/)-> . . . My::DB->join([qw/Distribution => modules /)-> . . .

Join comparison

• DBIC– rows belong to 1 table class

only– joined columns are either

• "side-products", • prefetched (all of them, no

choice)

– fixed join type

• DBIDM– rows inherit from all joined

tables– flattening of all columns

– default join type, can be overridden

Speed

12.04.23 - Page 1

DépartementOffice

Select speed

list mods(109349 rows)

join Auth dist mods(113895 rows)

DBI 0.43 secs 1.36 secs

DBIC regular 11.09 secs 15.50 secs

DBIC prefetch n.a. 146.29 secs

DBIC hashref inflator

10.06 secs 14.17 secs

DBIDM regular 4.00 secs 5.01 secs

DBIDM fast_statement

2.25 secs 3.28 secs

12.04.23 - Page 1

DépartementOffice

Insert speed

insert (72993 rows)

DBI 5.8 secs

DBIC regular 40.35 secs

DBIC populate 5.6 secs

DBIDM regular

32.81 secs

DBIDM bulk 32.57 secs

12.04.23 - Page 1

DépartementOffice

Classes and methods

DBIC : methods of a row

DB<1> m $auth # 152 methods_auth_id_accessor_cpanid_accessor_email_accessor_fullname_accessor_result_source_instance_accessoradd_to_distsauth_idcpaniddistsdists_rsemailfullnameresult_source_instancevia DBIx::Class::Core -> DBIx::Class::Relationship ->

DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasMany: has_many

via DBIx::Class::Core -> DBIx::Class::Relationship -> DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasOne: _get_primary_key

via DBIx::Class::Core -> DBIx::Class::Relationship -> DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasOne: _has_one

via DBIx::Class::Core -> DBIx::Class::Relationship -> DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasOne: _validate_has_one_condition

via DBIx::Class::Core -> DBIx::Class::Relationship -> DBIx::Class::Relationship::Helpers -> DBIx::Class::Relationship::HasOne: has_one

...

DB<4> x mro::get_linear_isa(ref $auth)0 ARRAY(0x13826c4) 0 'FPW12::DBIC::Result::Auth' 1 'DBIx::Class::Core' 2 'DBIx::Class::Relationship' 3 'DBIx::Class::Relationship::Helpers' 4 'DBIx::Class::Relationship::HasMany' 5 'DBIx::Class::Relationship::HasOne' 6 'DBIx::Class::Relationship::BelongsTo' 7 'DBIx::Class::Relationship::ManyToMany' 8 'DBIx::Class' 9 'DBIx::Class::Componentised' 10 'Class::C3::Componentised' 11 'DBIx::Class::AccessorGroup' 12 'Class::Accessor::Grouped' 13 'DBIx::Class::Relationship::Accessor' 14 'DBIx::Class::Relationship::CascadeActions' 15 'DBIx::Class::Relationship::ProxyMethods' 16 'DBIx::Class::Relationship::Base' 17 'DBIx::Class::InflateColumn' 18 'DBIx::Class::Row' 19 'DBIx::Class::PK::Auto' 20 'DBIx::Class::PK' 21 'DBIx::Class::ResultSourceProxy::Table' 22 'DBIx::Class::ResultSourceProxy'

DBIDM : methods of a row

DB<1> m $auth # 36 methodsdistsinsert_into_distsmetadmvia DBIx::DataModel::Source::Table:

_get_last_insert_idvia DBIx::DataModel::Source::Table:

_insert_subtreesvia DBIx::DataModel::Source::Table: _rawInsertvia DBIx::DataModel::Source::Table: _singleInsertvia DBIx::DataModel::Source::Table:

_weed_out_subtreesvia DBIx::DataModel::Source::Table: deletevia DBIx::DataModel::Source::Table:

has_invalid_columnsvia DBIx::DataModel::Source::Table: insertvia DBIx::DataModel::Source::Table: updatevia DBIx::DataModel::Source::Table ->

DBIx::DataModel::Source: TO_JSONvia DBIx::DataModel::Source::Table ->

DBIx::DataModel::Source: apply_column_handlervia DBIx::DataModel::Source::Table ->

DBIx::DataModel::Source: auto_expandvia DBIx::DataModel::Source::Table ->

DBIx::DataModel::Source: bless_from_DBvia DBIx::DataModel::Source::Table ->

DBIx::DataModel::Source: expandvia DBIx::DataModel::Source::Table ->

DBIx::DataModel::Source: fetch

DB<4> x mro::get_linear_isa(ref $auth)

0 ARRAY(0x145275c)

0 'FPW12::DBIDM::Auth'

1 'DBIx::DataModel::Source::Table'

2 'DBIx::DataModel::Source'

BelongsTo, HasMany, etc.

Schema Core ResultSet

My::DB My::DB::Result::Table_n

row resultset

DBIC classes

application classes

objects

schema

Class

Row

Componentised AccessorGroup

My::DB::ResultSet::Table_n

Relationship InflateColumn PK::Auto PK RSProxy::Table

BelongsTo, HasMany, etc.

RSProxyRLHelpers RLBase ResultSource

RS::Table

inheritanceinstantiation delegation

Schema

Source

Table Join Statement

My::DB My::DB::Table_n

My::DB::AutoJoin::

row statementrow

DBIDM classes

applicationclasses

objects

schema

DBIDM Meta-Architecture

Schema Table Join

My::DB

My::DB::Table_n

My::DB::Auto_join

Meta::Source

Meta::Table Meta::Join

meta::table

meta::join

meta::schema

Meta::Schema Meta::PathMeta::Association Meta::Type

meta::assoc meta::path

meta::type

DBIC introspection methods

• Schema– sources(), source($source_name)

• ResultSource– primary_key– columns(), column_info($column_name)– relationships(), relationship_info($relname)

Architecture comparison

• DBIC– very complex class structure– no distinction front/meta

layser– methods for

• CRUD• navigation to related objs• introspection• setting up columns• setting up relationships• …

• DBIDM– classes quite close to DBI

concepts– front classes and Meta

classes– methods for

• CRUD• navigation to related objs• access to meta

12.04.23 - Page 1

DépartementOffice

Resultset/Statement objects

12.04.23 - Page 1

DépartementOffice

Example task

• list names of authors – of distribs starting with 'DBIx' – and version number > 2

12.04.23 - Page 1

DépartementOffice

DBIC ResultSet chaining

my $dists = $schema->resultset('Dist')->search( {dist_name => {-like => 'DBIx%'}},);my $big_vers = $dists->search({dist_vers => { ">" => 2}});my @auths = $big_vers->search_related('auth', undef,

{distinct => 1, order_by => 'fullname'});

say $_->fullname foreach @auths;

# Magical join ! Magical "group by" !SELECT auth.auth_id, auth.email, auth.fullname, auth.cpanidFROM dists me JOIN auths auth ON auth.auth_id = me.auth_id WHERE ( ( dist_vers > ? AND dist_name LIKE ? ) ) GROUP BY auth.auth_id, auth.email, auth.fullname, auth.cpanid ORDER BY fullname

DBIDM Statement lifecycle

new

sqlized

prepared

executed

schema + source

data row(s)

new()

sqlize()

prepare()

execute()

bind()refine()

bind()

bind()

bind()execute()

next() / all()

blessedcolumn types applied

-post_bless

-pre_exec

-post_exec

-post_SQL

DBIDM Statement::refine()

my $stmt = FPW12::DBIDM->join(qw/Dist auth/)->select( -where => {dist_name => {-like => 'DBIx%'}}, -result_as => 'statement',);

$stmt->refine( -columns => [-DISTINCT => 'fullname'], -where => {dist_vers => { ">" => 0}}, -order_by => 'fullname',);

say $_->{fullname} while $_ = $stmt->next;

DBIDM subquery

my $stmt = FPW12::DBIDM::Dist->select( -where => {dist_name => {-like => 'DBIx%'}}, -result_as => 'statement',);

my $subquery = $stmt->select( -columns => 'auth_id', -where => {dist_vers => { ">" => 2}}, -result_as => 'subquery',);

my $auths = FPW12::DBIDM::Auth->select( -columns => 'fullname', -where => {auth_id => {-in => $subquery}},);

say $_->{fullname} foreach @$rows;

Statement comparison

• DBIC– powerful refinement

constructs• more "where" criteria• navigation to related source• column restriction• aggregation operators (e.g.

"count")

• DBIDM– limited refinement constructs– explicit control of status

12.04.23 - Page 1

DépartementOffice

Other features

12.04.23 - Page 1

DépartementOffice

Transactions

• DBIC

$schema->txn_do($sub);

– can be nested– can have intermediate

savepoints

• DBIDM

$sch->do_transaction($sub);

– can be nested– no intermediate savepoints

DBIC inflation/deflation

__PACKAGE__->inflate_column('insert_time', {inflate => sub { DateTime::Format::Pg->parse_datetime(shift); }, deflate => sub { DateTime::Format::Pg->format_datetime(shift) }, });

DBIDM Types (inflate/deflate)

# declare a TypeMy::DB->Type(Multivalue => from_DB => sub {$_[0] = [split /;/, $_[0]] }, to_DB => sub {$_[0] = join ";", @$_[0] },);

# apply it to some columns in a tableMy::DB::Author->metadm->define_column_type( Multivalue => qw/hobbies languages/,);

Extending/customizing DBIC

• Subclassing– result classes– resultset classes– storage– sqlmaker

Extending / customizing DBIDM

• Schema hooks for– SQL dialects (join syntax, alias syntax, limit / offset, etc.)– last_insert_id

• Ad hoc subclasses for– SQL::Abstract– Table– Join– Statements

• Statement callbacks• Extending table classes

– additional methods– redefining _singleInsert method

Not covered here

• inserts, updates, deletes• Schema generator• Schema versioning• inflation/deflation

12.04.23 - Page 1

DépartementOffice

THANK YOU FOR YOUR ATTENTION

12.04.23 - Page 1

DépartementOffice

Bonus slides

Why hashref instead of OO accessors ?

• Perl builtin rich API for hashes (keys, values, slices, string interpolation)

• good for import / export in YAML/XML/JSON• easier to follow steps in Perl debugger• faster than OO accessor methods• visually clear distinction between lvalue / rvalue

– my $val = $hashref->{column};– $hashref->{column} = $val;

• visually clear distinction between – $row->{column} / $row->remote_table()

SQL::Abstract::More : extensions

• -columns => [qw/col1|alias1 max(col2)|alias2/]– SELECT col1 AS alias1, max(col2) AS alias2

• -columns => [-DISTINCT => qw/col1 col2 col3/]– SELECT DISTINCT col1, col2, col3

• -order_by => [qw/col1 +col2 –col3/]– SELECT … ORDER BY col1, col2 ASC, col3 DESC

• -for => "update" || "read only"– SELECT … FOR UPDATE