magento indexes

Magento Indexers

Ivan ChepurnyiMagento Trainer / Lead Developer

Ivan Chepurnyi Magento Developers Meetup

Agenda

• Overview of Indexes Functionality• Creation of own indexes

Ivan Chepurnyi Magento Developers Meetup

Let Imagine…

… that Magento doesn’t have indexes:

• The prices in product list are calculated on the fly depending on catalog rules, tier prices for customer groups

• Stock availability for configurable and bundle products can be calculated only after loading the product collection

• Layered navigation data is build in real-time for product attributes information

• Anchor categories recursively collects subcategories for filtering product list

Ivan Chepurnyi Magento Developers Meetup

It’s all about performance…

The main goal is minimizing amount of operations to display

products to a customer

Ivan Chepurnyi Magento Developers Meetup

Definitions

• Indexed DataAggregated data for entity representation on the frontend lists.

• Indexer

Generates index data on event or manual by process.• Index Event

The moment when entity or related to it information is changed and that affects its index data.

• Index Process

Wrapper for indexer and contains information about its mode and status

• Main Controller

Forwards events to Index Process

Ivan Chepurnyi Magento Developers Meetup

Index Workflow

EventEventEvents

Process

Main Controller

Indexer

ManualInvoke

Indexed Data

Ivan Chepurnyi Magento Developers Meetup

Event Types

• Save

When indexed entity or related to it information was changed

• Delete

When indexed entity or related to it one was deleted

• Mass UpdateWhen batch of entities was updated. (Update Attributes on Product Grid)

Ivan Chepurnyi Magento Developers Meetup

Observed Entities

Indexed Entities– Product– Product Inventory– Category– Tag

Entities Scope– Customer Group– Website– Store Group– Store View

Ivan Chepurnyi Magento Developers Meetup

Index Process

Available Statuses

• Pending

Indicates that indexer is up to date

• Running

Index currently in process of full rebuilding index data.

• Require ReindexStatus for notifying admin user, that index is not up to date and should be rebuild.

Ivan Chepurnyi Magento Developers Meetup

Index Process

Indexing Modes• Real-time

• Manual

Event

EventRequire Reindex

Update Index Data

Ivan Chepurnyi Magento Developers Meetup

Indexer Flow

Match Event

Register Event Data

Reindex Data

Main C

ontroller

Index Process

Ivan Chepurnyi Magento Developers Meetup

Mage_Index Module

Main Controller

Process

Indexer Base

Mage_Index_Model_Indexer

Mage_Index_Model_Process

Mage_Index_Model_Indexer_Abstract

Ivan Chepurnyi Magento Developers Meetup

Catalog Module

Index Module

Indexers Modularity

Mage_Index_Model_Indexer_Abstract

Mage_Catalog_Model_Product_Indexer_Eav

Mage_Catalog_Model_Product_Indexer_Flat

Mage_Catalog_Model_Product_Indexer_Price

Inventory Module

Mage_CatalogIndex_Model_Indexer_Stock

…

Ivan Chepurnyi Magento Developers Meetup

ModelMage_Index_Model_Indexer_Abstract

Indexer Structure

Resource ModelMage_Index_Model_Mysql4_Abstract

Matches event data and runs appropriate method in resource model for re-indexing

Works directly with database for generation of the indexed data. Usually all the data operated via MySQL queries.

Ivan Chepurnyi Magento Developers Meetup

What can you use?

Mage_Index_Model_Indexer

getProcessByCode($indexerCode)

getProcessCollection()

processEntityAction($entity, $entityType, $eventType)

Mage_Index_Model_Process

reindexAll()

reindexEverything()

setMode($mode)

Ivan Chepurnyi Magento Developers Meetup

What you shouldn’t do…

• Invoke reindexAll method from index model/resource model, because it is better to let admin user know when the index was rebuild.

• Process entity events directly with indexer, instead of passing data through the main controller. You never know which index may depend on this event.

Ivan Chepurnyi Magento Developers Meetup

Creating own indexer

• Defining indexer in configuration• Designing index data table• Implementing model • Implementing resource model• Applying index on the frontend

Ivan Chepurnyi Magento Developers Meetup

Featured Products

There is easier way to create featured products functionality, but it is a simple example on what should be done for creation own indexer.

Ivan Chepurnyi Magento Developers Meetup

Defining index in configuration

<config>

<!-- …. module configurtaions -->

<global>

<!-- …. module configurtaions -->

<index>

<indexer>

<featured_products>

<model>your_module/indexer_featured</model>

</featured_products>

</indexer>

</index>

</global>

</config>

etc/config.xml

Indexer Model

Indexer Code

Ivan Chepurnyi Magento Developers Meetup

Designing index data table

• Adding new attribute to catalog product entity called is_featured

• Creating table that will contain product ids of products that are marked as featured products.

Ivan Chepurnyi Magento Developers Meetup

Designing index data table

$this->addAttribute('catalog_product', 'is_featured', array(

'type' => 'int',

'label' => 'Is featured',

'input' => 'select',

'source' => 'eav/entity_attribute_source_boolean',

'user_defined' => false,

'required' => false

));

Setup Script

Attribute Code

Yes/No Dropdown

Ivan Chepurnyi Magento Developers Meetup

Designing index data table

$table = new Varien_Db_Ddl_Table();

$table->setName($this->getTable(‘module/featured'));

$table->addColumn('product_id', Varien_Db_Ddl_Table::TYPE_INTEGER, 11, array(

'unsigned' => true,

'nullable' => false,

'primary' => true

));

$this->getConnection()->createTable($table);

Setup Script Table Alias

Table Column

Ivan Chepurnyi Magento Developers Meetup

Designing index data table

$table = new Varien_Db_Ddl_Table();

$table->setName($this->getTable(‘module/featured'));

$table->addColumn('product_id', Varien_Db_Ddl_Table::TYPE_INTEGER, 11, array(

'unsigned' => true,

'nullable' => false,

'primary' => true

));

$this->getConnection()->createTable($table);

Setup Script Table Alias

Table Column

Ivan Chepurnyi Magento Developers Meetup

Implementing Model

class Your_Module_Model_Indexer_Featured extends Mage_Index_Model_Indexer_Abstract

{

protected $_matchedEntities = array(

Mage_Catalog_Model_Product::ENTITY => array(

Mage_Index_Model_Event::TYPE_SAVE,

Mage_Index_Model_Event::TYPE_MASS_ACTION

)

);

}

Defining Matching Events

Entity Type

Event TypeEvent Types

Ivan Chepurnyi Magento Developers Meetup

Implementing Model

class Your_Module_Model_Indexer_Featured extends Mage_Index_Model_Indexer_Abstract

{

// … other code

protected function _construct()

{

$this->_init(‘your_module/indexer_featured');

}

}

Defining Indexer Resource Model

Resource model

Ivan Chepurnyi Magento Developers Meetup

Implementing Model

class Your_Module_Model_Indexer_Featured extends Mage_Index_Model_Indexer_Abstract

{

// … other code

public function getName()

{

return Mage::helper(‘your_module')->__('Featured Product');

}

public function getDescription()

{

return Mage::helper(‘‘your_module')->__('Indexes something');

}

}

Defining Indexer Information

Indexer Name in the admin

Indexer Description in the admin

Ivan Chepurnyi Magento Developers Meetup

Implementing Model

class Your_Module_Model_Indexer_Featured extends Mage_Index_Model_Indexer_Abstract

{

// … other code

protected function _registerEvent(Mage_Index_Model_Event $event)

{

/* @var $entity Mage_Catalog_Model_Product */

$entity = $event->getDataObject();

if ($entity->dataHasChangedFor('is_featured')) {

$event->setData('product_id', $entity->getId());

} elseif ($entity->getAttributesData()) {

$attributeData = $entity->getAttributesData();

if (isset($attributeData['is_featured'])) {

$event->setData('product_ids', $entity->getProductIds());

}

}

}

}

Register Event for Processing

Product Save Registering

Mass Action Registering

Ivan Chepurnyi Magento Developers Meetup

Implementing Model

class Your_Module_Model_Indexer_Featured extends Mage_Index_Model_Indexer_Abstract

{

// … other code

protected function _processEvent(Mage_Index_Model_Event $event)

{

if ($event->getData('product_id') || $event->getData('product_ids')) {

$this->callEventHandler($event);

}

}

}

Processing Event

Calling processor in resource model

Entity Type Event Type

catalogProductSave($event)

catalogProductMassAction($event)

Ivan Chepurnyi Magento Developers Meetup

Implementing Resource Model

class Your_Module_Model_Mysql4_Indexer_Featured extends Mage_Index_Model_Mysql4_Abstract

{

protected function _construct()

{

$this->_setResource(‘your_module');

}

}

Define resource connection

Your module resource prefix

Ivan Chepurnyi Magento Developers Meetup

Implementing Resource Model

class Your_Module_Model_Mysql4_Indexer_Featured extends Mage_Index_Model_Mysql4_Abstract

{

// … other code

protected function _reindexEntity($productId = null)

{

$select = $this->_getReadAdapter()->select();

/* @var $attribute Mage_Catalog_Model_Resource_Eav_Attribute */

$attribute = Mage::getSingleton('eav/config')

->getAttribute('catalog_product', 'is_featured');

$select->from($attribute->getBackendTable(), 'entity_id')

->where('value = ?', 1)

->where('attribute_id = ?', $attribute->getId());

Indexing Method

Retrieving only featured product ids

Ivan Chepurnyi Magento Developers Meetup

Implementing Resource Model

if ($productId !== null) {

if (!is_array($productId)) {

$productId = array($productId);

}

$select->where('entity_id IN(?)', $productId);

$this->_getWriteAdapter()->delete(

$this->getTable(‘your_module/featured'),

array(

'product_id IN(?)' => $productId

)

);

} else {

$this->_getWriteAdapter()->truncate($this->getTable(‘your_module/featured'));

}

Indexing MethodIf it is partial re-index, then delete

only related indexed data

Otherwise clear all indexed data

Ivan Chepurnyi Magento Developers Meetup

Implementing Resource Model

$sqlStatement = $select->insertIgnoreFromSelect(

$this->getTable(‘your_module/featured'),

array('product_id')

);

$this->_getWriteAdapter()->query($sqlStatement);

}

}

Indexing MethodFulfill index data from select we

created before

Ivan Chepurnyi Magento Developers Meetup

Implementing Resource Model

class Your_Module_Model_Mysql4_Indexer_Featured extends Mage_Index_Model_Mysql4_Abstract

{

// … other code

public function reindexAll()

{

$this->_reindexEntity();

}

}

Handling Events

Full index re-build

Ivan Chepurnyi Magento Developers Meetup

Implementing Resource Model

class Your_Module_Model_Mysql4_Indexer_Featured extends Mage_Index_Model_Mysql4_Abstract

{

// … other code

public function catalogProductSave($event)

{

$this->_reindexEntity($event->getData('product_id'));

}

public function catalogProductMassAction($event)

{

$this->_reindexEntity($event->getData('product_ids'));

}

}

Reindexing Events

Single Save Product Event

Mass Save Product Event

Ivan Chepurnyi Magento Developers Meetup

Applying Index for the frontend

• Observing and event catalog_product_collection_apply_limitations_after– Joining index table to product collection select– Create sub-select filter for collection

Ivan Chepurnyi Magento Developers Meetup

Liked it?

Checkout our advanced

training programs:http://www.ecomdev.org/magento-development-training-programs/advanced

Follow our blog posts:http://www.ecomdev.org/blog

Questions?