ancient to modern: upgrading nearly a decade of plone in public radio

Ancient To ModernUpgrading nearly a decade of Plone in Public Radio

!Image Credit: Fall Path by john22s CC-BY (https://www.flickr.com/photos/21926145@N04/15242844379)

this is the story of a journey

Of how a site born to Plone nearly a decade ago !Has grown with Plone ever since.

Image Credit: Amy Marbach “puppy snuggles are the best” CC-BY-NC-ND https://www.flickr.com/photos/amyofbadgroove/14327508940

Of how a site born to Plone nearly a decade ago !Has grown with Plone ever since.

Image Credit: Amy Marbach “puppy snuggles are the best” CC-BY-NC-ND https://www.flickr.com/photos/amyofbadgroove/14327508940

Image Credit: Richard Miles “Who Walks Who?…” CC-BY_NC_ND https://www.flickr.com/photos/_f1guy68_/2231052733

Of how a site born to Plone nearly a decade ago !Has grown with Plone ever since.

Image Credit: Tym “Great Strength” CC-BY-NC-ND https://www.flickr.com/photos/tym/269105455

It’s a success story for Plone, which plays to its greatest strengths

Image Credit: David Ellis CC-BY-NC-ND https://www.flickr.com/photos/97578613@N08/10771455296

Also a morality play about how incautious customizations can make upgrading a challenge.

The Beginning…

Plone 2.0

It started back in 2005, with Plone 2.0

The site had a nice, modern mid-aughts look and feel, with a clean color scheme and simple navigation.

It also featured drop-down menus for the many top-level navigation targets, for that easy-to-discover feel.

especially in areas where there were a large number of sub-items, like music shows in the “music” area.

Image Credit: Louise Docker CC-BY-NC-ND https://www.flickr.com/photos/aussiegall/6417891273

But like a lot of old things, it worked fine for its purpose

A new theme doesn’t justify an upgrade

There’s more!

But the look-and-feel were not the only issues

This method is called each time a “show” is viewed. !The highlighted line is preparing to run a query against an external RDBMS !This system was built to serve certain information from the Plone content to legacy PERL scripts, which were later abandoned.

This method is called each time a “show” is viewed. !The highlighted line is preparing to run a query against an external RDBMS !This system was built to serve certain information from the Plone content to legacy PERL scripts, which were later abandoned.

ZSQL scripts like this ran through much of the site functionality

* SQLWindowStorage to store some AT field data in an external database * Inefficient to retrieve this data field-by-field * Leads to “optimization” code that retrieves values all at once and stores them persistently in object attributes * Since this happens on __view__ of an object, there are lots of conflict errors and retries, meaning less efficient operations and more potential for real

errors.

* SQLWindowStorage to store some AT field data in an external database * Inefficient to retrieve this data field-by-field * Leads to “optimization” code that retrieves values all at once and stores them persistently in object attributes * Since this happens on __view__ of an object, there are lots of conflict errors and retries, meaning less efficient operations and more potential for real

errors.

* SQLWindowStorage to store some AT field data in an external database * Inefficient to retrieve this data field-by-field * Leads to “optimization” code that retrieves values all at once and stores them persistently in object attributes * Since this happens on __view__ of an object, there are lots of conflict errors and retries, meaning less efficient operations and more potential for real

errors.

* SQLWindowStorage to store some AT field data in an external database * Inefficient to retrieve this data field-by-field * Leads to “optimization” code that retrieves values all at once and stores them persistently in object attributes * Since this happens on __view__ of an object, there are lots of conflict errors and retries, meaning less efficient operations and more potential for real

errors.

Image Credit: http://failblog.cheezburger.com/thereifixedit http://failblog.cheezburger.com/thereifixedit

Clearly a safe and stable system for production. !

Image Credit: Todd Jordan “Wednesday’s Angry Boss” CC-BY-NC-ND https://www.flickr.com/photos/tojosan/14374817787

This raised the level of anger enough to really justify a large scale upgrade project.

Time for a Change

Plone?

But first, is Plone the right tool going forward?

Reasons To Stay• Heterogenous content in a deep tree • Broad editorial staff with different access

rights • In-Place management model • Staff familiarity • Transmogrifier

There were definitely reasons to stay

Upgrade or Migrate?

• Want to use dexterity types for new site • Heavy customizations • Lots of not-best-practice code • Better just to “start over”

Once the decision to stay was made, the next decision is to run an in-place upgrade, or a more extensive migration. !In this case, starting from scratch was really desirable, though the content needed to be preserved

Migration Out• Read content out from old Plone 2.5 site

• quintagroup.transmogrifier • Marshall as json

• collective.jsonify • Write site structure to disk

• modified q.transmogrifier writer

So a two-phase migration was planned, allowing us to transport the site content, stretching back into the late 1990s, into the new site.

A simple export pipeline moved the content to the filesystem as a series of folders and .json files

A simple export pipeline moved the content to the filesystem as a series of folders and .json files

A simple export pipeline moved the content to the filesystem as a series of folders and .json files

New Content Types• Dexterity Based • Custom shared behaviors

• IAirings (for things that go on air) • IScheduled (for things that occur on a

schedule) • IContentImages (for things that have a

standard set of images associated) • …

Next, time to set up the new content types. Using dexterity allowed us to build shared behaviors that would be used by different types.

Migration In• Read in json and map old types to new • Split pipeline for individual type handling • Remap locationsCreate new content

objects • Or, identify existing and update

(repeatable) • Reconnect related objects

With content types in place, the second phase involved migrating the content back into the new Plone site. !This process featured a number of the nice properties of the transmogrifier system.

The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content. !You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take. !A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects in the new system. !This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests and artists.

The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content. !You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take. !A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects in the new system. !This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests and artists.

The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content. !You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take. !A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects in the new system. !This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests and artists.

The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content. !You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take. !A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects in the new system. !This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests and artists.

The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content. !You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take. !A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects in the new system. !This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests and artists.

The “splitter” pipeline section allows you to set up individual pipelines associated with some subset of your migrating content. !You can predicate entry to a particular pipeline on any number of tests, but in this case we used the portal type to determine the path to take. !A second feature is the ability to annotate the transmogrifier object itself with values like a mapping of user ids in the old system to UIDs for user objects in the new system. !This mapping can then be shared from other pipeline sections that need to re-connect radio shows with their hosts and producers, or episodes with guests and artists.

36 hours later

Finally, we were able to run the migration. !An advantage of the generator-based approach of transmogrifier is that although the migration took 36 hours, and moved well over 300,000 objects, including images fetched over HTTP, the memory usage was constant and the server had little problem completing the job.

Now we have content!

And now we have plone content, but it isn’t all that great looking.

Make it pretty!

Image Credit: Kevin O’Mara “I Feel Pretty” https://www.flickr.com/photos/kevinomara/13394685405

The client engaged a well-known design firm in NYC, Hard Candy Shell

stylish, modern design !fully responsive, changing UI elements size and positioning to fit the device.

stylish, modern design !fully responsive, changing UI elements size and positioning to fit the device.

stylish, modern design !fully responsive, changing UI elements size and positioning to fit the device.

The mockups featured a lot of “block” content !With the same types of items appearing in different ways depending on location and purpose !HTML consists of a lot of repeating fragments showing up in various places

The mockups featured a lot of “block” content !With the same types of items appearing in different ways depending on location and purpose !HTML consists of a lot of repeating fragments showing up in various places

The mockups featured a lot of “block” content !With the same types of items appearing in different ways depending on location and purpose !HTML consists of a lot of repeating fragments showing up in various places

Other Requirements

• Some pages had to be “composable” • collective.cover

• Some pages would be pre-built, but feature dynamic content selection

• custom browserviews & templates • We want to keep our code DRY

zope.contentprovider

• Much like a browserview, but for a fragment • Multi-adapters of context, request and view • Allows you to render some object differently

due to the view in which it is being seen. • Add named adapters, and you have four

possible axes for choosing a rendering

These requirements led us to investigate a little-known zope package that powers the Plone portlet system.

Content Providers

A content provider is and adapter, initializing it works much like its cousin the browser view. Then, when called, it is updated via an “update” method and then rendered to an HTML fragment via the template. !Once a generic version is set up, making a custom version can be as simple as subclassing, giving it a different name, and providing a different template.

Content Providers

A content provider is and adapter, initializing it works much like its cousin the browser view. Then, when called, it is updated via an “update” method and then rendered to an HTML fragment via the template. !Once a generic version is set up, making a custom version can be as simple as subclassing, giving it a different name, and providing a different template.

In Cover Tiles

In cover tiles, we created a “content provider” tile, which was essentially identical to the basic persistent tile from collective.cover. !The big difference was an added “content_provider” method, which resolved a name and passed that and the other required arguments into a “content_provider” module-level function. !This method was then called from the template for all “tiles” in the project. !The pass-through to the module-level function simply ran the standard content provider rendering chain, but allowed for a name-based resolution of which provider to render.

In Cover Tiles

In cover tiles, we created a “content provider” tile, which was essentially identical to the basic persistent tile from collective.cover. !The big difference was an added “content_provider” method, which resolved a name and passed that and the other required arguments into a “content_provider” module-level function. !This method was then called from the template for all “tiles” in the project. !The pass-through to the module-level function simply ran the standard content provider rendering chain, but allowed for a name-based resolution of which provider to render.

In Cover Tiles

In cover tiles, we created a “content provider” tile, which was essentially identical to the basic persistent tile from collective.cover. !The big difference was an added “content_provider” method, which resolved a name and passed that and the other required arguments into a “content_provider” module-level function. !This method was then called from the template for all “tiles” in the project. !The pass-through to the module-level function simply ran the standard content provider rendering chain, but allowed for a name-based resolution of which provider to render.

Picture of KCRW homepage

This let us build features like this row on the live homepage, which contains two shows and one episode, managed by KCRW staff.

In Browserviews

We can use the same basic trick in a browser view. First we add a “content_provider” method to the view code, then we call that code from the view template, passing in any arguments we need to customize each call. Finally, that same method renders a contextually appropriate provider, and returns an HTML fragment for the page to use.

In Browserviews

We can use the same basic trick in a browser view. First we add a “content_provider” method to the view code, then we call that code from the view template, passing in any arguments we need to customize each call. Finally, that same method renders a contextually appropriate provider, and returns an HTML fragment for the page to use.

In Browserviews

We can use the same basic trick in a browser view. First we add a “content_provider” method to the view code, then we call that code from the view template, passing in any arguments we need to customize each call. Finally, that same method renders a contextually appropriate provider, and returns an HTML fragment for the page to use.

In Browserviews

We can use the same basic trick in a browser view. First we add a “content_provider” method to the view code, then we call that code from the view template, passing in any arguments we need to customize each call. Finally, that same method renders a contextually appropriate provider, and returns an HTML fragment for the page to use.

Picture of HR show homepage

This style allowed us to create sections like these “featured content” blocks on the music area and individual show landing pages. Here, individual items gathered by a view method are individually rendered with contextually appropriate HTML, creating a dynamic and flexible page.

Picture of HR show homepage

This style allowed us to create sections like these “featured content” blocks on the music area and individual show landing pages. Here, individual items gathered by a view method are individually rendered with contextually appropriate HTML, creating a dynamic and flexible page.

Outcomes

• Using the same content provider let us write templates once, use anywhere

• Having default names let us establish a standard display and then custom ones

• Adapting catalog brains as context let us avoid expensive .getObject() calls

In general, the outcomes of this approach were positive. Early investments in flexibility paid off as later additions “just worked”, with minimal updates.

Quirks• HCS design used a custom grid system • All rows shared the same markup • But cell markup differed depending on

how many in a row • Not convenient for dynamic content • Required a custom grid engine for

collective.cover • Would have been nice to have input in

that decision

But there were issues. For example, in service of the responsive design the custom grid provided by HCS put the responsibility for awareness of layout on cells within a row. From a programming perspective, this inversion of responsibility caused a bit of a headache. In retrospect I would recommend having a member of the development team on board in the theme planning process to allow for better coordination of approach.

Happily, collective.cover allows for customized grid layout engines, and we built one with a special case for every possible configuration of the layout. !

Ugly, but it works

But It Works

No, it wasn’t pretty, but it worked.

Other Features• Adapters provide consistent API for

shared behavior • Views expose these APIs • Client-side .js plugins consume and

display the data • Mobile app also consumes the same

apis

shared behavior apis: get latest airing get airings between x and y get upcoming things get playable media for this thing get data about media to be played

For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches for various API methods. !The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used in javascript plugins to power the download flyout menus that appear throughout the site.

For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches for various API methods. !The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used in javascript plugins to power the download flyout menus that appear throughout the site.

For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches for various API methods. !The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used in javascript plugins to power the download flyout menus that appear throughout the site.

For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches for various API methods. !The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used in javascript plugins to power the download flyout menus that appear throughout the site.

For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches for various API methods. !The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used in javascript plugins to power the download flyout menus that appear throughout the site.

For example, the IPlayable adapter. The interface defined a number of properties and methods. An abstract implementation provided methods that could be coded the same for all content types. Then individual adapters for various content types, including catalog brains, allowed for customized approaches for various API methods. !The approach allows us to use a single API on various content types to provide information like download links for playable media files. These can be used in javascript plugins to power the download flyout menus that appear throughout the site.

Other Features• Javascript-based player for live or

recorded audio • Player can be broken out into a stand-

alone window • Viewed on mobile, player can play when

browser app is closed • Player is persistent across page loads

The HCS designs called for an in-browser media player. One challenge was allowing playback in this player to persist across page loads as the user browsed the site.

By extending the powerful history.js plugin, we were able to ajaxify links, forms and other page-loading elements in the design. We were also able to include calls to google analytics, and reload all dynamic front-end features so that from the user’s perspective the site experience is smooth and unsurprising.

By extending the powerful history.js plugin, we were able to ajaxify links, forms and other page-loading elements in the design. We were also able to include calls to google analytics, and reload all dynamic front-end features so that from the user’s perspective the site experience is smooth and unsurprising.

By extending the powerful history.js plugin, we were able to ajaxify links, forms and other page-loading elements in the design. We were also able to include calls to google analytics, and reload all dynamic front-end features so that from the user’s perspective the site experience is smooth and unsurprising.

Other Features

• Solr integration provides great search results (alm.solrindex)

• Content from external WP blogs is regularly indexed and searchable

• Content_provider approach helps theme external content distinctively

A final feature was integration of the solr text search engine. We are now wrapping up a new feature that will index external blog content from KCRWs extensive WordPress blogiverse into their Plone site. This allows in-site search to surface external content alongside the content managed inside the site.

A celery-managed task runs daily to collect the blogs named in the site. !This fires off individual celery tasks to read and index the content from each blog. !The tasks result in the temporary creation of objects for each posts that live just long enough to be indexed into the catalog. !Then, our content providers allow us to represent these posts in a distinct way within search results.

A celery-managed task runs daily to collect the blogs named in the site. !This fires off individual celery tasks to read and index the content from each blog. !The tasks result in the temporary creation of objects for each posts that live just long enough to be indexed into the catalog. !Then, our content providers allow us to represent these posts in a distinct way within search results.

A celery-managed task runs daily to collect the blogs named in the site. !This fires off individual celery tasks to read and index the content from each blog. !The tasks result in the temporary creation of objects for each posts that live just long enough to be indexed into the catalog. !Then, our content providers allow us to represent these posts in a distinct way within search results.

A celery-managed task runs daily to collect the blogs named in the site. !This fires off individual celery tasks to read and index the content from each blog. !The tasks result in the temporary creation of objects for each posts that live just long enough to be indexed into the catalog. !Then, our content providers allow us to represent these posts in a distinct way within search results.

Many Thanks To• KCRW, who gave us the job • Alec Mitchell, who led the project • HCS, for the wonderful and feature-rich

design • The creators and maintainers of the add-

ons we used • PloneConf for the chance to show it all off

And that’s all we have time for. There’s much more to cover, but at this point I’ll have to stop. !Before I do, thanks to all the folks involved in this project. It was enormously fun to build and satisfying to see in action.

And You!

Any questions?

And You!

questions?

Any questions?