You are here: Home Blog Plone Content migration using transmogrifier and collective.jsonify
Personal tools

Plone Content migration using transmogrifier and collective.jsonify

Posted by daniel at Jun 17, 2014 11:30 AM |
A little known Plone product called collective.jsonify made our content migration a whole lot easier on several recent projects.

Introduction

Ok, you want to migrate some valuable Plone content from an old site to a new one.  You have, broadly speaking two options, though you may not be aware of the second one:

  1. Use Plone's migration tool & follow the Plone Upgrade Guide
  2. Use Transmogrifier and/or Jsonmigrator

Which you choose is up to you, but my approach is to use option 1 for upgrades from Plone 4.0 upwards, and sometimes Plone 3.3.x, I would use option 2 for anything earlier.

Upgrade Alternatives - Pros & Cons

As I see it the pros & cons are as follows:

  • The migration tool is the officially supported route, so you can be more entitled to expect support, however if you are upgrading from a long way back then you may find it to be a long process, if you need to upgrade to intermediate versions. Depending on the addons you previously used, you may also end up with broken products which don't upgrade well, or even at all - potentially leaving you needing to fix them yourself for modern Plone versions.  But, if you are upgrading from Plone 3.3.x or later, with few or no addons then you might be well advised to try this before the alternative below.
  • Transmogrifier and Jsonmigrator may take longer for small upgrades (eg from 3.3.6 upwards), but they enable you to start with a clean Plone 4 site & pull data in.  You can make switches from Archetypes to Dexterity based content types quite easily. And it is possibly to migrate data from outdated products into newer equivalents. Broken products should not haunt you!  However, you may have to write migration pipelines yourself for some more complex situations/products. Also I am unable to comment yet on migrating multi-lingual content or content history. I also skip importing workflow history, though that may be possible.  That being said, I haven't got stuck yet!

Common Transmogrifier stories

Once you have decided to use Transmogrifier there are 3 ways to use Transmogrifier to do migrations/content imports:

  1. Export from the old site to files, then import from the files to the new site, as documented by QuintaGroup.  This is alright, but it is less practical on older versions of Plone as you need to be able to install Transmogrifier on the old site as well as the new one & this gets difficult (or maybe impossible) before Plone buildouts were introduced.
  2. Install collective.jsonify on the old site to present the data in JSON format & pull it into the new site using collective.jsonmigrator.  This can be installed on some very old Plone sites - or even just plain Zope sites (I've done as far back as Plone 2.5 or Zope 2.6.4). This is the subject of the remainder of this article.
  3. Use Funnelweb to scrape the site HTML - this is useful for pulling in data from non-Plone sites, but more work for Plone sites than the solutions above, as you have to provide XPaths for each content field.

Get going with Jsonmigrator

Ok, that was a long introduction, so let's just get down to business.

Let's agree from the outset that we won't be silly about it.  Secondly, lets agree that it would be silly to try to export directly from the old site (especially since what we do here will open up security loopholes, like allowing unauthorized access to your entire site);  and that's without mentioning the obvious fact that crawling  your old site would potentially slow it down quite considerably.

  1. Right, so that means the first thing we do is to replicate the old site in a protected environment - build it however you built the old one & backup the old one, and transfer the data in. Let's call this duplicate the "Source" site from now on.
  2. Add collective.jsonify and simplejson to your "Source" Plone.
    To check if you need simplejson, run zopepy and "import simplejson" - if you get an ImportError then you do.
    $ bin/zopepy
    >>> import simplejson
    ImportError: No module named simplejson
    >>>

    To install them do one of the following options.  At the time of writing the collective.jsonify egg on Pypi doesn't have the latest changes, but you are free to use the builds here at your own risk (!):

    1. Add them to your eggs section of your (old site's) buildout (this uses my 0.1.8b build at the release area mentioned above):

      find-links = 
           ...
           https://github.com/djowett/collective.jsonify/archive/0.1.8b.zip
      
      eggs =
          Plone
          Pillow
          lxml
          project.policy
      # Temporarily added for transmogrifier 
          simplejson
          collective.jsonify
      
    2. If you don't have a buildout on your "Source" site, maybe because it's too old, then you will probably be best adding collective.jsonify to your Plone's python using one of the method's described here.
  3. Create and add a file named json_methods.py in <ploneinstance>/parts/instance/Extensions/ with the following content (create the Extensions folder if it doesn't exist)
    from collective.jsonify import get_item
    from collective.jsonify import get_children

  4. Go to the root of the ZMI and add 2 external methods for each of the above, like so:

    Add External Method
  5. Now test this by accessing the <Plone site>/get_children and <Plone site>/get_item.
  6. Add collective.jsonmigrator to the target Plone buildout (this will also pull in collective.transmogrifier and plone.app.transmogrifier).  If you check it out as a development egg, then it will be easier to tweak the transmogrifier pipeline when you need to.
    So add the highlighted lines to your develop.cfg buildout file (again, note that this uses my branch of jsonmigrator):
    [sources]
    ...
    collective.jsonmigrator = git https://github.com/collective/collective.jsonmigrator.git
    
    [buildout]
    eggs +=
    ...
        collective.jsonmigrator
    
  7. Rebuild your target plone (probably with an empty DB), set up the new Plone site and go to <plone>/@@jsonmigrator
  8. Select the pipeline from the dropdown list relevant to the version of Plone (or Zope) on your "Source" site and whether you want to extract everything or just the results of a "catalog query" from your Source site. 
  9. Now tweak the settings on the next page, and see how you get on.
  10. Extra changes will need you to edit the relevant pipeline file in your Target site under: <Plone Site>/src/collective.jsonmigrator/collective/jsonmigrator/<pipeline-name>.cfg
  11. Finally, in order to see some details correctly (like creation & modification dates) you will probably need to rebuild the Portal Catalog through the ZMI > portal_catalog > Advanced > "Clear and Rebuild".
  12. Have fun!

Other useful links on the subject from:

 

If you want more help, then please get in touch through our contact form.