The document discusses Transmogrifier, a tool for migrating content in Plone. It describes what Transmogrifier is, provides a simple example of its configuration, and shows how to create custom blueprints for processing data. It also discusses lessons learned from large migrations and ideas for future improvements.

1. Migrations with Transmogrifier Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento

2. What is transmogrifier? Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento original intent was to provide pluggable way to import content into plone Migrations with Transmogrifier help us construct pipeline , from ordered sections (blueprints), where each section proccesses data in pipe can be also used for other tasks that require data proccessing tools. nice example which shows power of transmogrifier is collective.funnelweb

3. Simple Example of Config Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento [transmogrifier] pipeline = source transform 1 transform 2 create [source] blueprint = example. transmogrifier .source size = 5 [transform 1] blueprint = example.transmogrifier.transform from_field = text to_ field = text transform = rst condition = python:item['filename'].endswith('.rst') [transform 2] blueprint = example.transmogrifier.transform from_field = text to_ field = description transform = get_first_paragraph [create] blueprint = collective.transmogrifier.constructor Migrations with Transmogrifier

4. Create your own blueprint /1 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento class ExampleSource(object): classProvides(ISectionBlueprint) implements(ISection) def __init__(self, transmogrifier, name, options, previous): self.previous = previous self.name = name def __iter__(self): for item in self.previous: yield item for record in ReadJSONFile(): yield dict(_path=normalize(record['title']), _type=”Document”, title =record['title'], text=record['text']) Migrations with Transmogrifier

5. Create your own blueprint /2 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento from collective.transmogrifier.utils import Condition class ExampleTranform(object): classProvides(ISectionBlueprint) implements(ISection) def __init__(self, transmogrifier, name, options, previous): self.previous = previous self.from_ field = options['from_field'] self.to_ field = options['to_field'] self.tranform = options['tranform'] self.condition = Condition(condition, transmogrifier, name, options) def __iter__(self): for item in self.previous: if self.condition(item): if 'rst' == self.param: item[self.to_field] = TransformFromRSTToHTML(item[self.from_field ]) elif 'get_first_paragraph' == self.param: item[self.to_field] = GetFirstParagraph(item[self.to_field]) yield item Migrations with Transmogrifier

6. Register custom Blueprint Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento <configure xmlns=&quot;http://namespaces.zope.org/zope&quot; xmlns:tg=&quot;http://namespaces.plone.org/transmogrifier&quot; i18n_domain=&quot;example.transmogrifier&quot;> <tg:registerConfig name=&quot;exampleconfig&quot; title=&quot;Example&quot; description=&quot;Example pipeline config&quot; configuration=&quot; example .cfg&quot; /> <utility component=&quot;.source.ExampleSource&quot; name=&quot;example.transmogrifier.source&quot; /> <utility component=&quot;.transform.ExampleTransform&quot; name=&quot;example.transmogrifier.transform&quot; /> </configure> Migrations with Transmogrifier

7. Base Blueprints /1 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento collective.transmogrifier.sections.constructor Migrations with Transmogrifier collective.transmogrifier.sections.savepoint contstructs plone content (_type, _path) commits a savepoint which as result free up memory collective.transmogrifier.sections.savepoint commits a savepoint which as result free up memory collective.transmogrifier.sections.csvsource reads csv file and puts lines of it in pipeline collective.transmogrifier.sections.splitter splits pipeline, which lets you proccess data flow separatly

8. Base Blueprints /2 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento collective.transmogrifier.sections.manipulator Migrations with Transmogrifier rename, copy, delete data in pipeline collective.transmogrifier.sections.condition Lets you discard data from pipeline collective.transmogrifier.sections.inserter Lets you create new fields for data (from other fields) collective.transmogrifier.sections.codec Changes encoding for data in pipeline collective.transmogrifier.sections.constructor Creates parent folders when item needs them

9. Some Useful Bluprints Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento plone.app.transmogrifier Migrations with Transmogrifier collective.funellweb transmogrify.siteanalyser transmogrify.filesystem transmogrify.ploneremote transmogrify.webcrawler transmogrify.htmltesting transmogrify.htmlcontentextractor transmogrify.pathsorter quintagroup.transmogrifier collective.blueprint.dancing transmogrify.sqlalchemy collective.blueprint.translationlinker

10. Plone 2.0 > Plone 4.0 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento >100.000 content objects Migrations with Transmogrifier >40GB in size many custom products usual migration/upgrade failed

11. Lessons Learned Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento make migration repeatable (eg: using GenericSetup upgrade steps) Migrations with Transmogrifier migrate parts of page separatly split migration into export and import part monitor and time migration don't complicate in making blueprints extra reusable

12. Plone 2.0 > Plone 4.0 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento Migrations with Transmogrifier <10GB of “Data.fs” (RelStorage) + 20GB of blobs less custom code, better re-usage of Plone's default features using less resources, noticably faster thanks to Plone4 happy client

13. Whats missing? Wishlist!? Future! Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento Migrations with Transmogrifier Better support for monitoring what goes inside pipeline Web GUI for base blueprints Making it not dependable on plone

14. Thank you Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento Migrations with Transmogrifier Rok Garbas, rok@garbas.si http://www.garbas.si http://www.twitter.com/garbas