SlideShare une entreprise Scribd logo
1  sur  34
Télécharger pour lire hors ligne
Migrate
Getting it into Drupal
   andrew morton
drewish@zivtech.com
Migrate 2.0
•   Powerful object oriented framework for
    moving content into Drupal.
•   Minimal UI, primarily code based.
•   Steep learning curve (aka migraine
    module) but hopefully this talk will help.
Migrate 2.0
•   Drupal 6 requires autoload and dbtng
    modules. So the code is very similar in 6
    and 7.
•   Migrate Extras provides support for many
    contrib modules.
•   The best documentation is in the wine.inc
    and beer.inc example code.
Why not just use
         Feeds?
•   If it does what you need, use it. It’s much
    easier to setup.
•   Migrate is faster, and more flexible but
    you need to write code to map fields.
•   Feeds doesn’t work well if you’ve got
    different content types that need to
    reference each other.
The Concepts
•   Source
•   Destination
•   Map
•   Field Mapping
•   Field Handlers
•   Migration
Source
•   Interface to your existing data (SQL, CSV,
    XML, JSON)
•   Provides a list of fields and descriptions
•   Iterates (reset, next) over rows
Destination
•   Interface for writing data to Drupal—
    typically to a Entity.
•   Creates one record for each record in the
    source. If you’re creating users and
    profiles, you’ll need two migrations.
Map
        Source                     Destination
SQL, XML, JSON, CSV, etc          Node, User, Term, etc
   ID    Name     Age            entity_id   field_name   field_age

   1     Larry    34                32        Larry          34
   2     Curly    54                33        Curly          54
   4      Moe     47                34         Moe           47


                           Map
                  Source ID   Dest ID
                      1         32
                      2         33
                      4         34
Map
•   Connects the source’s ID to the
    destination’s ID.
•   Provides lookup facilities.
•   Allows created items to be deleted as
    part of a rollback.
Field Mappings
        Source                         Destination
SQL, XML, JSON, CSV, etc              Node, User, Term, etc
   ID   Name     Age    Junk         entity_id   field_name   field_age

   1    Larry    34     blah            32        Larry          34
   2    Curly    54                     33        Curly          54
   4     Moe     47     Spam            34         Moe           47



                 Field Mappings
                       Source   Destination
                        Name    field_name
                        Age      field_age
                        Junk       NULL
Field Mappings
•   Links a source field to a destination field.
•   Lets you look up IDs from the other
    migration’s maps with sourceMigration().
•   If the destination has a complex structure
    (e.g. Address or file field) then additional
    data is passed in via arguments().
Migration
•   Code that glues the parts together.
•   In your constructor you setup: source,
    destination, map and field mappings.
•   Allows customization at several points:
    prepareRow(), prepare(), complete().
Field Handlers
•   Handles the details of converting the field
    into the structure that Drupal
    understands.
•   Turns $entity->field_bar   = “foo”   into
    $entity->field_bar[‘und’][0][‘value’] = “foo”

•   Might pull additional data out of
    arguments.
Destination Handler
•   Provides additional functionality to a
    destination, e.g. comment adding a field
    to nodes for comment status.
•   Destination delegates calls to fields(),
    prepare(), complete() to the destination
    handlers.
•   You probably won’t need to write one.
UI Demo
•   Lets go look at the UI.
Drush Commands
•   migrate-status (ms) - List all migrations and
    display their current status.

•   migrate-import (mi) - Start a migration and
    create/update destination objects.

•   migrate-reset-status (mrs) - Reset a migration.

•   migrate-rollback (mr) - Delete a migration’s
    destination objects.
Basic Module
•   Create a new module
•   Implement hook_migrate_api()
•   Create a class that extends Migration and
    setup the source, destination, map and
    field mappings in the constructor
•   Register the class in the .info file
SQL Source
// inside __construct()

$query = db_select('migrate_example_beer_topic',
'met')
  ->fields('met', array('style', 'details',
    'style_parent', 'region', 'hoppiness'))
  ->orderBy('style_parent', 'ASC');

$this->source = new MigrateSourceSQL($query);
Or a CSV Source
// The definition of the columns. Keys are integers,
// values are an array of field name then description.
$columns = array(
  0 => array('cvs_uid', 'Id'),
  1 => array('email', 'Email'),
  2 => array('name', 'Name'),
  3 => array('date', 'Date'),
);

// Instantiate the class using the path to the CSV
// file and the columns.
$path = 'path/relative/to/drupal/root/your_file.csv';
$this->source = new MigrateSourceCSV($path, $columns);
Other Sources
•   There are also classes for importing
    content from XML and JSON.
•   Lots of variation among sources so expect
    to do some tweaking.
Source Base Classes
•   If you can fetch IDs separately from values:
    •   Use MigrateSourceList as a source
    •   Implement MigrateList for fetching counts and
        IDs, and MigrateItem for fetching values
•   If everything is in a single file with IDs mixed in:
    •   Use MigrateSourceMultiItems as a source
    •   Implement MigrateItems for extracting IDs and
        values
Migration Map
// inside __construct()

$this->map = new MigrateSQLMap($this->machineName,
   array(
      'style' => array(
        'type' => 'varchar',
        'length' => 255,
        'not null' => TRUE,
        'description' => 'Topic ID',
      )
   ),
   MigrateDestinationTerm::getKeySchema()
);
Destinations
// inside __construct()

// Create terms...
$this->destination = new
  MigrateDestinationTerm('example_beer_styles');

// ...or nodes...
$this->destination = new
  MigrateDestinationNode('article');

// ...or
$this->destination = new
  MigrateDestinationUser();
Creating Destinations
•   Hopefully you won’t need to.
•   If you’re working with entities created by
    the Entity API make sure you look at:
    http://drupal.org/node/1168196
Field Mappings
// inside __construct()

// Can be as simple as this...
$this->addFieldMapping('name', 'style');

// ...or more complicated.
$this->addFieldMapping(NULL, 'region')
  ->description('This is broken')
  ->issueGroup(t('Client Issues'))
  ->issuePriority(
    MigrateFieldMapping::ISSUE_PRIORITY_MEDIUM)
  ->issueNumber(770064);
Field Mapping
           Arguments
•   Generally used as a hack to pass multiple
    source fields into a single destination
    field.
•   Use this magic syntax to have arguments
    replaced by values from the source row:
    $this->addFieldMapping('D', 'S1')->arguments(
       array('A1' => array('source_field' => 'S2')),
       array('A2' => array('source_field' => 'S3'))
    );
Field Mapping
         Arguments
// Files have so many arguments there’s a helper
// to build the array:
$arguments = MigrateFileFieldHandler::arguments(
  drupal_get_path('module', 'migrate_example'),
  'file_copy', FILE_EXISTS_RENAME, NULL,
  array('source_field' => 'image_alt'),
  array('source_field' => 'image_description'));
$this->addFieldMapping('field_image', 'image')
  ->arguments($arguments);
Field Mapping
    Source Migrations
•   When you have an ID value from the old
    system and need to look up the new ID
    from the Migration Map:
    $this->addFieldMapping('uid', 'author_id')
      ->sourceMigration('BeerUser')

•   Add a dependency to make sure the
    other migration runs first:
    $this->dependencies = array('BeerUser');

•
Circular
        Dependencies
•   Break them by using stubs.
•   Implement createStub($migration), create
    a dummy record and return the new id.
•   Specify a sourceMigration on the field
    mapping.
prepareRow($row)
•   Passes in the source row as an object so
    you can make modifications.
•   Add or change field values by modifying
    the properties:
    $row->name = $row->first . “ ” . $row->last;
    $row->created = strtotime($row->access);

•   Return FALSE to indicate that rows
    should be skipped over during an import.
prepare($entity, $row)
•   Passes in the entity object with properties
    populated by field mappings, and the
    source row.
•   Last chance to make changes before the
    entity is saved.
•   If you have an unsupported field type you
    can manually populate it here:
complete($ent, $row)
•   Passes in the saved entity (with any ID
    values from auto increment fields) and
    the source row.
•   This is the place if you need to update
    other records to reference the new entity.
•   If you’re doing a node_save($ent) in here,
    you’re doing it wrong.
Writing a Field
          Handler
•   Hopefully you won’t need to.
•   Register compatible field type in
    constructor.
•   Handle conversion in prepare().
•   Optionally, use complete() for follow
    tasks.
Thanks!
Any Questions?

Contenu connexe

Tendances

Patterns for slick database applications
Patterns for slick database applicationsPatterns for slick database applications
Patterns for slick database applicationsSkills Matter
 
Entities in drupal 7
Entities in drupal 7Entities in drupal 7
Entities in drupal 7Zsolt Tasnadi
 
Drupal Entities - Emerging Patterns of Usage
Drupal Entities - Emerging Patterns of UsageDrupal Entities - Emerging Patterns of Usage
Drupal Entities - Emerging Patterns of UsageRonald Ashri
 
Drupal - dbtng 25th Anniversary Edition
Drupal - dbtng 25th Anniversary EditionDrupal - dbtng 25th Anniversary Edition
Drupal - dbtng 25th Anniversary Editionddiers
 
Why async and functional programming in PHP7 suck and how to get overr it?
Why async and functional programming in PHP7 suck and how to get overr it?Why async and functional programming in PHP7 suck and how to get overr it?
Why async and functional programming in PHP7 suck and how to get overr it?Lucas Witold Adamus
 
Scala bad practices, scala.io 2019
Scala bad practices, scala.io 2019Scala bad practices, scala.io 2019
Scala bad practices, scala.io 2019Loïc Knuchel
 
Scala for Java Programmers
Scala for Java ProgrammersScala for Java Programmers
Scala for Java ProgrammersEric Pederson
 
Advanced Django ORM techniques
Advanced Django ORM techniquesAdvanced Django ORM techniques
Advanced Django ORM techniquesDaniel Roseman
 
11. session 11 functions and objects
11. session 11   functions and objects11. session 11   functions and objects
11. session 11 functions and objectsPhúc Đỗ
 
Type safe embedded domain-specific languages
Type safe embedded domain-specific languagesType safe embedded domain-specific languages
Type safe embedded domain-specific languagesArthur Xavier
 
Objective-C for iOS Application Development
Objective-C for iOS Application DevelopmentObjective-C for iOS Application Development
Objective-C for iOS Application DevelopmentDhaval Kaneria
 
Marc’s (bio)perl course
Marc’s (bio)perl courseMarc’s (bio)perl course
Marc’s (bio)perl courseMarc Logghe
 
Drupal Field API. Practical usage
Drupal Field API. Practical usageDrupal Field API. Practical usage
Drupal Field API. Practical usagePavel Makhrinsky
 
Indexing and Query Optimizer (Richard Kreuter)
Indexing and Query Optimizer (Richard Kreuter)Indexing and Query Optimizer (Richard Kreuter)
Indexing and Query Optimizer (Richard Kreuter)MongoDB
 
Mongoseattle indexing-2010-07-27
Mongoseattle indexing-2010-07-27Mongoseattle indexing-2010-07-27
Mongoseattle indexing-2010-07-27MongoDB
 

Tendances (20)

Patterns for slick database applications
Patterns for slick database applicationsPatterns for slick database applications
Patterns for slick database applications
 
Your Entity, Your Code
Your Entity, Your CodeYour Entity, Your Code
Your Entity, Your Code
 
Entities in drupal 7
Entities in drupal 7Entities in drupal 7
Entities in drupal 7
 
Drupal Entities - Emerging Patterns of Usage
Drupal Entities - Emerging Patterns of UsageDrupal Entities - Emerging Patterns of Usage
Drupal Entities - Emerging Patterns of Usage
 
Drupal - dbtng 25th Anniversary Edition
Drupal - dbtng 25th Anniversary EditionDrupal - dbtng 25th Anniversary Edition
Drupal - dbtng 25th Anniversary Edition
 
Why async and functional programming in PHP7 suck and how to get overr it?
Why async and functional programming in PHP7 suck and how to get overr it?Why async and functional programming in PHP7 suck and how to get overr it?
Why async and functional programming in PHP7 suck and how to get overr it?
 
ORM in Django
ORM in DjangoORM in Django
ORM in Django
 
Scala bad practices, scala.io 2019
Scala bad practices, scala.io 2019Scala bad practices, scala.io 2019
Scala bad practices, scala.io 2019
 
Scala for Java Programmers
Scala for Java ProgrammersScala for Java Programmers
Scala for Java Programmers
 
Build your own entity with Drupal
Build your own entity with DrupalBuild your own entity with Drupal
Build your own entity with Drupal
 
Advanced Django ORM techniques
Advanced Django ORM techniquesAdvanced Django ORM techniques
Advanced Django ORM techniques
 
11. session 11 functions and objects
11. session 11   functions and objects11. session 11   functions and objects
11. session 11 functions and objects
 
Django Pro ORM
Django Pro ORMDjango Pro ORM
Django Pro ORM
 
Type safe embedded domain-specific languages
Type safe embedded domain-specific languagesType safe embedded domain-specific languages
Type safe embedded domain-specific languages
 
Objective-C for iOS Application Development
Objective-C for iOS Application DevelopmentObjective-C for iOS Application Development
Objective-C for iOS Application Development
 
Marc’s (bio)perl course
Marc’s (bio)perl courseMarc’s (bio)perl course
Marc’s (bio)perl course
 
Drupal Field API. Practical usage
Drupal Field API. Practical usageDrupal Field API. Practical usage
Drupal Field API. Practical usage
 
Field api.From d7 to d8
Field api.From d7 to d8Field api.From d7 to d8
Field api.From d7 to d8
 
Indexing and Query Optimizer (Richard Kreuter)
Indexing and Query Optimizer (Richard Kreuter)Indexing and Query Optimizer (Richard Kreuter)
Indexing and Query Optimizer (Richard Kreuter)
 
Mongoseattle indexing-2010-07-27
Mongoseattle indexing-2010-07-27Mongoseattle indexing-2010-07-27
Mongoseattle indexing-2010-07-27
 

En vedette

GeorgiaGov's move to Drupal - presentation by Nikhil Deshpande @nikofthehill
GeorgiaGov's move to Drupal - presentation by Nikhil Deshpande @nikofthehillGeorgiaGov's move to Drupal - presentation by Nikhil Deshpande @nikofthehill
GeorgiaGov's move to Drupal - presentation by Nikhil Deshpande @nikofthehillNikhil Deshpande
 
Building Archivable Websites
Building Archivable WebsitesBuilding Archivable Websites
Building Archivable Websitesnullhandle
 
Drush 5.0 (DrupalCamp LA 2012) - Chris Charlton
Drush 5.0 (DrupalCamp LA 2012) - Chris CharltonDrush 5.0 (DrupalCamp LA 2012) - Chris Charlton
Drush 5.0 (DrupalCamp LA 2012) - Chris CharltonChris Charlton
 
Staging Drupal 8 31 09 1 3
Staging Drupal 8 31 09 1 3Staging Drupal 8 31 09 1 3
Staging Drupal 8 31 09 1 3Drupalcon Paris
 
Create Website In Indian Languages using drupal
Create Website In Indian Languages using drupalCreate Website In Indian Languages using drupal
Create Website In Indian Languages using drupaldrupalindia
 
Drupal Migration
Drupal MigrationDrupal Migration
Drupal Migration永对 陈
 
Using Drupal Features in B-Translator
Using Drupal Features in B-TranslatorUsing Drupal Features in B-Translator
Using Drupal Features in B-TranslatorDashamir Hoxha
 
8 Web Practices for Drupal
8  Web Practices for Drupal8  Web Practices for Drupal
8 Web Practices for DrupalWingston
 
Moving In: how to port your content from * to Drupal
Moving In: how to port your content from * to DrupalMoving In: how to port your content from * to Drupal
Moving In: how to port your content from * to DrupalEmma Jane Hogbin Westby
 
Moving Drupal to the Cloud
Moving Drupal to the CloudMoving Drupal to the Cloud
Moving Drupal to the CloudAri Davidow
 
Drupal in the Cloud with Windows Azure
Drupal in the Cloud with Windows AzureDrupal in the Cloud with Windows Azure
Drupal in the Cloud with Windows AzureFord AntiTrust
 
Data migration to Drupal using the migrate module
Data migration to Drupal using the migrate moduleData migration to Drupal using the migrate module
Data migration to Drupal using the migrate moduleLuc Bézier
 
Migraine Drupal - syncing your staging and live sites
Migraine Drupal - syncing your staging and live sitesMigraine Drupal - syncing your staging and live sites
Migraine Drupal - syncing your staging and live sitesdrupalindia
 
Content Staging in Drupal 8
Content Staging in Drupal 8Content Staging in Drupal 8
Content Staging in Drupal 8Dick Olsson
 
Managing Translation Workflows in Drupal 7
Managing Translation Workflows in Drupal 7Managing Translation Workflows in Drupal 7
Managing Translation Workflows in Drupal 7Suzanne Dergacheva
 
Migration from Legacy CMS to Drupal
Migration from Legacy CMS to DrupalMigration from Legacy CMS to Drupal
Migration from Legacy CMS to DrupalRachel Jaro
 

En vedette (17)

GeorgiaGov's move to Drupal - presentation by Nikhil Deshpande @nikofthehill
GeorgiaGov's move to Drupal - presentation by Nikhil Deshpande @nikofthehillGeorgiaGov's move to Drupal - presentation by Nikhil Deshpande @nikofthehill
GeorgiaGov's move to Drupal - presentation by Nikhil Deshpande @nikofthehill
 
Recipes for Drupal distributions
Recipes for Drupal distributionsRecipes for Drupal distributions
Recipes for Drupal distributions
 
Building Archivable Websites
Building Archivable WebsitesBuilding Archivable Websites
Building Archivable Websites
 
Drush 5.0 (DrupalCamp LA 2012) - Chris Charlton
Drush 5.0 (DrupalCamp LA 2012) - Chris CharltonDrush 5.0 (DrupalCamp LA 2012) - Chris Charlton
Drush 5.0 (DrupalCamp LA 2012) - Chris Charlton
 
Staging Drupal 8 31 09 1 3
Staging Drupal 8 31 09 1 3Staging Drupal 8 31 09 1 3
Staging Drupal 8 31 09 1 3
 
Create Website In Indian Languages using drupal
Create Website In Indian Languages using drupalCreate Website In Indian Languages using drupal
Create Website In Indian Languages using drupal
 
Drupal Migration
Drupal MigrationDrupal Migration
Drupal Migration
 
Using Drupal Features in B-Translator
Using Drupal Features in B-TranslatorUsing Drupal Features in B-Translator
Using Drupal Features in B-Translator
 
8 Web Practices for Drupal
8  Web Practices for Drupal8  Web Practices for Drupal
8 Web Practices for Drupal
 
Moving In: how to port your content from * to Drupal
Moving In: how to port your content from * to DrupalMoving In: how to port your content from * to Drupal
Moving In: how to port your content from * to Drupal
 
Moving Drupal to the Cloud
Moving Drupal to the CloudMoving Drupal to the Cloud
Moving Drupal to the Cloud
 
Drupal in the Cloud with Windows Azure
Drupal in the Cloud with Windows AzureDrupal in the Cloud with Windows Azure
Drupal in the Cloud with Windows Azure
 
Data migration to Drupal using the migrate module
Data migration to Drupal using the migrate moduleData migration to Drupal using the migrate module
Data migration to Drupal using the migrate module
 
Migraine Drupal - syncing your staging and live sites
Migraine Drupal - syncing your staging and live sitesMigraine Drupal - syncing your staging and live sites
Migraine Drupal - syncing your staging and live sites
 
Content Staging in Drupal 8
Content Staging in Drupal 8Content Staging in Drupal 8
Content Staging in Drupal 8
 
Managing Translation Workflows in Drupal 7
Managing Translation Workflows in Drupal 7Managing Translation Workflows in Drupal 7
Managing Translation Workflows in Drupal 7
 
Migration from Legacy CMS to Drupal
Migration from Legacy CMS to DrupalMigration from Legacy CMS to Drupal
Migration from Legacy CMS to Drupal
 

Similaire à Migrate

Jooctrine - Doctrine ORM in Joomla!
Jooctrine - Doctrine ORM in Joomla!Jooctrine - Doctrine ORM in Joomla!
Jooctrine - Doctrine ORM in Joomla!Herman Peeren
 
How and Where in GLORP
How and Where in GLORPHow and Where in GLORP
How and Where in GLORPESUG
 
Graph Databases in the Microsoft Ecosystem
Graph Databases in the Microsoft EcosystemGraph Databases in the Microsoft Ecosystem
Graph Databases in the Microsoft EcosystemMarco Parenzan
 
Elasticsearch and Symfony Integration - Debarko De
Elasticsearch and Symfony Integration - Debarko DeElasticsearch and Symfony Integration - Debarko De
Elasticsearch and Symfony Integration - Debarko DeDebarko De
 
Fields in Core: How to create a custom field
Fields in Core: How to create a custom fieldFields in Core: How to create a custom field
Fields in Core: How to create a custom fieldIvan Zugec
 
Schema Design (Mongo Austin)
Schema Design (Mongo Austin)Schema Design (Mongo Austin)
Schema Design (Mongo Austin)MongoDB
 
Java/Scala Lab: Борис Трофимов - Обжигающая Big Data.
Java/Scala Lab: Борис Трофимов - Обжигающая Big Data.Java/Scala Lab: Борис Трофимов - Обжигающая Big Data.
Java/Scala Lab: Борис Трофимов - Обжигающая Big Data.GeeksLab Odessa
 
MongoDB & NoSQL 101
 MongoDB & NoSQL 101 MongoDB & NoSQL 101
MongoDB & NoSQL 101Jollen Chen
 
Data Migration with Spark to Hive
Data Migration with Spark to HiveData Migration with Spark to Hive
Data Migration with Spark to HiveDatabricks
 
Dive into spark2
Dive into spark2Dive into spark2
Dive into spark2Gal Marder
 
Code is not text! How graph technologies can help us to understand our code b...
Code is not text! How graph technologies can help us to understand our code b...Code is not text! How graph technologies can help us to understand our code b...
Code is not text! How graph technologies can help us to understand our code b...Andreas Dewes
 
ElasticSearch AJUG 2013
ElasticSearch AJUG 2013ElasticSearch AJUG 2013
ElasticSearch AJUG 2013Roy Russo
 
PofEAA and SQLAlchemy
PofEAA and SQLAlchemyPofEAA and SQLAlchemy
PofEAA and SQLAlchemyInada Naoki
 
Solr 6 Feature Preview
Solr 6 Feature PreviewSolr 6 Feature Preview
Solr 6 Feature PreviewYonik Seeley
 
Using the latest Java Persistence API 2 Features - Tech Days 2010 India
Using the latest Java Persistence API 2 Features - Tech Days 2010 IndiaUsing the latest Java Persistence API 2 Features - Tech Days 2010 India
Using the latest Java Persistence API 2 Features - Tech Days 2010 IndiaArun Gupta
 
Migrating data into Drupal using the migrate module
Migrating data into Drupal using the migrate moduleMigrating data into Drupal using the migrate module
Migrating data into Drupal using the migrate moduleJohan Gant
 

Similaire à Migrate (20)

Migrate
MigrateMigrate
Migrate
 
Drupal 8: Fields reborn
Drupal 8: Fields rebornDrupal 8: Fields reborn
Drupal 8: Fields reborn
 
Jooctrine - Doctrine ORM in Joomla!
Jooctrine - Doctrine ORM in Joomla!Jooctrine - Doctrine ORM in Joomla!
Jooctrine - Doctrine ORM in Joomla!
 
How and Where in GLORP
How and Where in GLORPHow and Where in GLORP
How and Where in GLORP
 
Graph Databases in the Microsoft Ecosystem
Graph Databases in the Microsoft EcosystemGraph Databases in the Microsoft Ecosystem
Graph Databases in the Microsoft Ecosystem
 
Elasticsearch and Symfony Integration - Debarko De
Elasticsearch and Symfony Integration - Debarko DeElasticsearch and Symfony Integration - Debarko De
Elasticsearch and Symfony Integration - Debarko De
 
Fields in Core: How to create a custom field
Fields in Core: How to create a custom fieldFields in Core: How to create a custom field
Fields in Core: How to create a custom field
 
Schema Design (Mongo Austin)
Schema Design (Mongo Austin)Schema Design (Mongo Austin)
Schema Design (Mongo Austin)
 
Java/Scala Lab: Борис Трофимов - Обжигающая Big Data.
Java/Scala Lab: Борис Трофимов - Обжигающая Big Data.Java/Scala Lab: Борис Трофимов - Обжигающая Big Data.
Java/Scala Lab: Борис Трофимов - Обжигающая Big Data.
 
MongoDB & NoSQL 101
 MongoDB & NoSQL 101 MongoDB & NoSQL 101
MongoDB & NoSQL 101
 
Data Migration with Spark to Hive
Data Migration with Spark to HiveData Migration with Spark to Hive
Data Migration with Spark to Hive
 
Dive into spark2
Dive into spark2Dive into spark2
Dive into spark2
 
Code is not text! How graph technologies can help us to understand our code b...
Code is not text! How graph technologies can help us to understand our code b...Code is not text! How graph technologies can help us to understand our code b...
Code is not text! How graph technologies can help us to understand our code b...
 
ElasticSearch AJUG 2013
ElasticSearch AJUG 2013ElasticSearch AJUG 2013
ElasticSearch AJUG 2013
 
Full metal mongo
Full metal mongoFull metal mongo
Full metal mongo
 
PofEAA and SQLAlchemy
PofEAA and SQLAlchemyPofEAA and SQLAlchemy
PofEAA and SQLAlchemy
 
Solr 6 Feature Preview
Solr 6 Feature PreviewSolr 6 Feature Preview
Solr 6 Feature Preview
 
Using the latest Java Persistence API 2 Features - Tech Days 2010 India
Using the latest Java Persistence API 2 Features - Tech Days 2010 IndiaUsing the latest Java Persistence API 2 Features - Tech Days 2010 India
Using the latest Java Persistence API 2 Features - Tech Days 2010 India
 
Sparklyr
SparklyrSparklyr
Sparklyr
 
Migrating data into Drupal using the migrate module
Migrating data into Drupal using the migrate moduleMigrating data into Drupal using the migrate module
Migrating data into Drupal using the migrate module
 

Plus de Zivtech, LLC

Shift left testing for continuous collaboration
Shift left testing for continuous collaborationShift left testing for continuous collaboration
Shift left testing for continuous collaborationZivtech, LLC
 
DrupalCon San Francisco- The State of Drupal as a Web Application & Product ...
DrupalCon San Francisco-  The State of Drupal as a Web Application & Product ...DrupalCon San Francisco-  The State of Drupal as a Web Application & Product ...
DrupalCon San Francisco- The State of Drupal as a Web Application & Product ...Zivtech, LLC
 
Probo.ci Drupal 4 Gov Devops 1/2 day Presentation
Probo.ci Drupal 4 Gov Devops 1/2 day Presentation Probo.ci Drupal 4 Gov Devops 1/2 day Presentation
Probo.ci Drupal 4 Gov Devops 1/2 day Presentation Zivtech, LLC
 
Drupaldelphia 2014 - Mission Bicycle Case Study - Slides by Matt Cheney
Drupaldelphia 2014 - Mission Bicycle Case Study - Slides by Matt CheneyDrupaldelphia 2014 - Mission Bicycle Case Study - Slides by Matt Cheney
Drupaldelphia 2014 - Mission Bicycle Case Study - Slides by Matt CheneyZivtech, LLC
 
Open Source isn't Just Good, it's Good Business - DrupalCamp Colorado 2014
Open Source isn't Just Good, it's Good Business - DrupalCamp Colorado 2014Open Source isn't Just Good, it's Good Business - DrupalCamp Colorado 2014
Open Source isn't Just Good, it's Good Business - DrupalCamp Colorado 2014Zivtech, LLC
 
Drupal Problem-Solving Techniques
Drupal Problem-Solving TechniquesDrupal Problem-Solving Techniques
Drupal Problem-Solving TechniquesZivtech, LLC
 
The business case for contributing code
The business case for contributing codeThe business case for contributing code
The business case for contributing codeZivtech, LLC
 
Css compass sasssusy
Css   compass sasssusyCss   compass sasssusy
Css compass sasssusyZivtech, LLC
 
From a Contractor to a Shop: How to grow your Drupal business without losing ...
From a Contractor to a Shop: How to grow your Drupal business without losing ...From a Contractor to a Shop: How to grow your Drupal business without losing ...
From a Contractor to a Shop: How to grow your Drupal business without losing ...Zivtech, LLC
 
Drupal Presentation @ the Higher Education Web Symposium
Drupal Presentation @ the Higher Education Web SymposiumDrupal Presentation @ the Higher Education Web Symposium
Drupal Presentation @ the Higher Education Web SymposiumZivtech, LLC
 

Plus de Zivtech, LLC (10)

Shift left testing for continuous collaboration
Shift left testing for continuous collaborationShift left testing for continuous collaboration
Shift left testing for continuous collaboration
 
DrupalCon San Francisco- The State of Drupal as a Web Application & Product ...
DrupalCon San Francisco-  The State of Drupal as a Web Application & Product ...DrupalCon San Francisco-  The State of Drupal as a Web Application & Product ...
DrupalCon San Francisco- The State of Drupal as a Web Application & Product ...
 
Probo.ci Drupal 4 Gov Devops 1/2 day Presentation
Probo.ci Drupal 4 Gov Devops 1/2 day Presentation Probo.ci Drupal 4 Gov Devops 1/2 day Presentation
Probo.ci Drupal 4 Gov Devops 1/2 day Presentation
 
Drupaldelphia 2014 - Mission Bicycle Case Study - Slides by Matt Cheney
Drupaldelphia 2014 - Mission Bicycle Case Study - Slides by Matt CheneyDrupaldelphia 2014 - Mission Bicycle Case Study - Slides by Matt Cheney
Drupaldelphia 2014 - Mission Bicycle Case Study - Slides by Matt Cheney
 
Open Source isn't Just Good, it's Good Business - DrupalCamp Colorado 2014
Open Source isn't Just Good, it's Good Business - DrupalCamp Colorado 2014Open Source isn't Just Good, it's Good Business - DrupalCamp Colorado 2014
Open Source isn't Just Good, it's Good Business - DrupalCamp Colorado 2014
 
Drupal Problem-Solving Techniques
Drupal Problem-Solving TechniquesDrupal Problem-Solving Techniques
Drupal Problem-Solving Techniques
 
The business case for contributing code
The business case for contributing codeThe business case for contributing code
The business case for contributing code
 
Css compass sasssusy
Css   compass sasssusyCss   compass sasssusy
Css compass sasssusy
 
From a Contractor to a Shop: How to grow your Drupal business without losing ...
From a Contractor to a Shop: How to grow your Drupal business without losing ...From a Contractor to a Shop: How to grow your Drupal business without losing ...
From a Contractor to a Shop: How to grow your Drupal business without losing ...
 
Drupal Presentation @ the Higher Education Web Symposium
Drupal Presentation @ the Higher Education Web SymposiumDrupal Presentation @ the Higher Education Web Symposium
Drupal Presentation @ the Higher Education Web Symposium
 

Migrate

  • 1. Migrate Getting it into Drupal andrew morton drewish@zivtech.com
  • 2. Migrate 2.0 • Powerful object oriented framework for moving content into Drupal. • Minimal UI, primarily code based. • Steep learning curve (aka migraine module) but hopefully this talk will help.
  • 3. Migrate 2.0 • Drupal 6 requires autoload and dbtng modules. So the code is very similar in 6 and 7. • Migrate Extras provides support for many contrib modules. • The best documentation is in the wine.inc and beer.inc example code.
  • 4. Why not just use Feeds? • If it does what you need, use it. It’s much easier to setup. • Migrate is faster, and more flexible but you need to write code to map fields. • Feeds doesn’t work well if you’ve got different content types that need to reference each other.
  • 5. The Concepts • Source • Destination • Map • Field Mapping • Field Handlers • Migration
  • 6. Source • Interface to your existing data (SQL, CSV, XML, JSON) • Provides a list of fields and descriptions • Iterates (reset, next) over rows
  • 7. Destination • Interface for writing data to Drupal— typically to a Entity. • Creates one record for each record in the source. If you’re creating users and profiles, you’ll need two migrations.
  • 8. Map Source Destination SQL, XML, JSON, CSV, etc Node, User, Term, etc ID Name Age entity_id field_name field_age 1 Larry 34 32 Larry 34 2 Curly 54 33 Curly 54 4 Moe 47 34 Moe 47 Map Source ID Dest ID 1 32 2 33 4 34
  • 9. Map • Connects the source’s ID to the destination’s ID. • Provides lookup facilities. • Allows created items to be deleted as part of a rollback.
  • 10. Field Mappings Source Destination SQL, XML, JSON, CSV, etc Node, User, Term, etc ID Name Age Junk entity_id field_name field_age 1 Larry 34 blah 32 Larry 34 2 Curly 54 33 Curly 54 4 Moe 47 Spam 34 Moe 47 Field Mappings Source Destination Name field_name Age field_age Junk NULL
  • 11. Field Mappings • Links a source field to a destination field. • Lets you look up IDs from the other migration’s maps with sourceMigration(). • If the destination has a complex structure (e.g. Address or file field) then additional data is passed in via arguments().
  • 12. Migration • Code that glues the parts together. • In your constructor you setup: source, destination, map and field mappings. • Allows customization at several points: prepareRow(), prepare(), complete().
  • 13. Field Handlers • Handles the details of converting the field into the structure that Drupal understands. • Turns $entity->field_bar = “foo” into $entity->field_bar[‘und’][0][‘value’] = “foo” • Might pull additional data out of arguments.
  • 14. Destination Handler • Provides additional functionality to a destination, e.g. comment adding a field to nodes for comment status. • Destination delegates calls to fields(), prepare(), complete() to the destination handlers. • You probably won’t need to write one.
  • 15. UI Demo • Lets go look at the UI.
  • 16. Drush Commands • migrate-status (ms) - List all migrations and display their current status. • migrate-import (mi) - Start a migration and create/update destination objects. • migrate-reset-status (mrs) - Reset a migration. • migrate-rollback (mr) - Delete a migration’s destination objects.
  • 17. Basic Module • Create a new module • Implement hook_migrate_api() • Create a class that extends Migration and setup the source, destination, map and field mappings in the constructor • Register the class in the .info file
  • 18. SQL Source // inside __construct() $query = db_select('migrate_example_beer_topic', 'met') ->fields('met', array('style', 'details', 'style_parent', 'region', 'hoppiness')) ->orderBy('style_parent', 'ASC'); $this->source = new MigrateSourceSQL($query);
  • 19. Or a CSV Source // The definition of the columns. Keys are integers, // values are an array of field name then description. $columns = array(   0 => array('cvs_uid', 'Id'),   1 => array('email', 'Email'),   2 => array('name', 'Name'),   3 => array('date', 'Date'), ); // Instantiate the class using the path to the CSV // file and the columns. $path = 'path/relative/to/drupal/root/your_file.csv'; $this->source = new MigrateSourceCSV($path, $columns);
  • 20. Other Sources • There are also classes for importing content from XML and JSON. • Lots of variation among sources so expect to do some tweaking.
  • 21. Source Base Classes • If you can fetch IDs separately from values: • Use MigrateSourceList as a source • Implement MigrateList for fetching counts and IDs, and MigrateItem for fetching values • If everything is in a single file with IDs mixed in: • Use MigrateSourceMultiItems as a source • Implement MigrateItems for extracting IDs and values
  • 22. Migration Map // inside __construct() $this->map = new MigrateSQLMap($this->machineName, array( 'style' => array( 'type' => 'varchar', 'length' => 255, 'not null' => TRUE, 'description' => 'Topic ID', ) ), MigrateDestinationTerm::getKeySchema() );
  • 23. Destinations // inside __construct() // Create terms... $this->destination = new MigrateDestinationTerm('example_beer_styles'); // ...or nodes... $this->destination = new MigrateDestinationNode('article'); // ...or $this->destination = new MigrateDestinationUser();
  • 24. Creating Destinations • Hopefully you won’t need to. • If you’re working with entities created by the Entity API make sure you look at: http://drupal.org/node/1168196
  • 25. Field Mappings // inside __construct() // Can be as simple as this... $this->addFieldMapping('name', 'style'); // ...or more complicated. $this->addFieldMapping(NULL, 'region') ->description('This is broken') ->issueGroup(t('Client Issues')) ->issuePriority( MigrateFieldMapping::ISSUE_PRIORITY_MEDIUM) ->issueNumber(770064);
  • 26. Field Mapping Arguments • Generally used as a hack to pass multiple source fields into a single destination field. • Use this magic syntax to have arguments replaced by values from the source row: $this->addFieldMapping('D', 'S1')->arguments( array('A1' => array('source_field' => 'S2')), array('A2' => array('source_field' => 'S3')) );
  • 27. Field Mapping Arguments // Files have so many arguments there’s a helper // to build the array: $arguments = MigrateFileFieldHandler::arguments( drupal_get_path('module', 'migrate_example'), 'file_copy', FILE_EXISTS_RENAME, NULL, array('source_field' => 'image_alt'), array('source_field' => 'image_description')); $this->addFieldMapping('field_image', 'image') ->arguments($arguments);
  • 28. Field Mapping Source Migrations • When you have an ID value from the old system and need to look up the new ID from the Migration Map: $this->addFieldMapping('uid', 'author_id') ->sourceMigration('BeerUser') • Add a dependency to make sure the other migration runs first: $this->dependencies = array('BeerUser'); •
  • 29. Circular Dependencies • Break them by using stubs. • Implement createStub($migration), create a dummy record and return the new id. • Specify a sourceMigration on the field mapping.
  • 30. prepareRow($row) • Passes in the source row as an object so you can make modifications. • Add or change field values by modifying the properties: $row->name = $row->first . “ ” . $row->last; $row->created = strtotime($row->access); • Return FALSE to indicate that rows should be skipped over during an import.
  • 31. prepare($entity, $row) • Passes in the entity object with properties populated by field mappings, and the source row. • Last chance to make changes before the entity is saved. • If you have an unsupported field type you can manually populate it here:
  • 32. complete($ent, $row) • Passes in the saved entity (with any ID values from auto increment fields) and the source row. • This is the place if you need to update other records to reference the new entity. • If you’re doing a node_save($ent) in here, you’re doing it wrong.
  • 33. Writing a Field Handler • Hopefully you won’t need to. • Register compatible field type in constructor. • Handle conversion in prepare(). • Optionally, use complete() for follow tasks.