upfront diet for plone

Roché Compaan

About me

● Owner of Upfront Systems● Founded Upfront Systems in 1998● Zope developer for the past 10 years

Experience

● Largest Plone deployments to date: 5FM & MetroFM● 5FM

– 20 million page views per month– 70 thousand members– 6 million objects in ZODB

● Metro FM– 15 million page views per month– 20 thousand members– 2 million objects in ZODB

● Helping Connexions with performance● http://www.markhorner.net/2010/04/29/connexions-authoring-

performance/

Can Plone scale to 10 million documents?

ZODB Benchmarks

● Most common myth spread: “ZODB is a high read, (s)low write database”

● No published ZODB benchmarks● Wrote some benchmarks that test ZODB

read/write speed in 2008● collective.zodbbench

ZODB Write Speed

0 500 thousand 2,5 million 5 million 7,5 million 8,5 million 10 million0

1000

2000

3000

4000

5000

6000

7000

8000

Postgres

ZODB

Number of records

Inse

rts/

seco

nd

ZODB Lookup Speed

Number of Objects Lookup Time100,000 0.003 ms

1,000,000 0.006 ms10,000,000 2 ms

Benchmark Conclusions

● Thanks Jim, ZODB Rocks! ● Large cache size important for writes too, not

only reads● Need a solution for more than 10 million objects● If the ZODB can handle 250 inserts/second at

peak volume why does it take a full minute to add 250 ATDocuments?

A closer look at Plone

● Added 10,000 documents to Plone, while profiling with Dieter Maurer's ZopeProfiler

● Timed “apply_index” method to find indexes performing badly with and without experimental.catalogqueryplan

● Batch import of content with an without references.

● Used fsdump to inspect pickle contents.

Plone Benchmark Conclusions

● portal_catalog remains a bottleneck to scaling a Plone site.

● 1 catalog == many many conflicts● reference_catalog is a major performance

bottleneck● Indexing 10,000 documents shows that 90% of

the objects in Data.fs relate to indexing.

What's in a pickle?

2,51755

81,564

9,872

20,02485

655

1,544

30,455

182,708

7798,088122

Pickle contents of a single transaction adding an ATDocument

object size in bytes

IIBTree

Folder

IOBucket

OIBucket

IIBucket

IOBTree

BTrees.Length

ATDocument

OOBTree

IISet

Archetypes.BaseUnit

OOBucket

IITreeSet

Think before you index!

“Waking up objects is expensive” - this is an irrational fear. And adding another catalog in

response to this fear requires therapy.

The portal_catalog problem

● No navigation, portal tabs, folder contents, review queue if your content type is not in the portal_catalog.

● Your content type is automatically subscribed to 20+ indexes in the portal_catalog. Bad for new developers with hopes of building an enterprise CMS.

● Unless you have 5+ years experience with Zope – it's not so easy to “opt out”

● portal_catalog should only be used for content searches – no functionality must depend on it.

What is upfront.diet?

● It's a healthier and happier approach to Plone development: index only when you really really need to.

● Set of packages that re-implement plone functionality without depending on the portal_catalog and reference_catalog.

● Includes replacements for: portal tabs, navigation portlet, folder contents, archetype references.

● Provides a catalog blacklist/whitelist to customise indexing per content type.

More about upfront.diet

● upfront.navportlet● Separate catalog for navportlet

● upfront.simplereferencefield● Store references directly on object using annotations● still depends uid_catalog

● upfront.foldercontents● Adapter for IFolderish● Index allowedRolesAndUsers directly on folder – less conflicts

● upfront.catalogblacklist● Generic Setup profile with a whitelist and blacklist of indexes per

content type● patches IndexableObjectWrapper, ugh!

upfront.diet todo list

● Packages:● Portal tabs (done but not a separate package yet)● Sitemap● Review queue● News and Events Smart Folders● Site Search

● References without uid_catalog● Comparative benchmarks with and without it

upfront.diet deployments

● Very few at the moment but looks promising● Fullmarks.org.za (half a million objects, Data.fs

< 1GB)

Contact Me

● Email: roche@upfrontsystems.co.za● Blog:

http://upfrontsystems.co.za/Members/roche● facebook.com/rochecompaan● twitter.com/rochecompaan

Questions?