happy together: creating successful magento erp integrations | imagine 2013…
TRANSCRIPT
![Page 1: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/1.jpg)
Monday, April 8, 13
![Page 2: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/2.jpg)
Creating SuccessfulMagento ERP Integrations
Monday, April 8, 13
![Page 3: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/3.jpg)
Happy TogetherCreating Successful Magento ERP Integrations
CTO / Lead Engineerwww.classyllama.com
David Alger
Monday, April 8, 13
![Page 4: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/4.jpg)
A Little About Me
• Exclusively focused on Magento eCommerce since early ’09• Member of the Magento Technical Partner Council• Member of the Magento Certification Advisory Board• Magento Certified Developer Plus• Magento Front End Certified Developer• Zend Certified Engineer• Experienced Software Integrator
Monday, April 8, 13
![Page 5: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/5.jpg)
Architecting and engineering a Magento ERP integration while avoiding the pitfalls and challenges that come with integrating
multiple enterprise-level software components
Creating Magento ERP Integrations
Monday, April 8, 13
![Page 6: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/6.jpg)
What’s On Our Agenda
• Reasons for Integrating• Purpose Driven Nature• Architectural Design
• Facing Challenges• Engineering Solutions• Integration Scalability
Monday, April 8, 13
![Page 7: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/7.jpg)
Why an Integration?• An ERP in simple terms:
– Software built around the premise of having the ability to manage all of the information and functions of a company seamlessly and in real-time
• What benefits are there?– Centralization, accuracy, less hassle!
• Order Fulfillment• Customer Service• Product Management
• Integrating Magento with your ERP means an increase in efficiency
Monday, April 8, 13
![Page 8: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/8.jpg)
Purpose Driven Nature• Integrations with any software should be purpose driven• What are the goals of the integration?
– Example: Guaranteed accurate information across properties– Example: Less headaches during order fulfillment
Monday, April 8, 13
![Page 9: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/9.jpg)
Architectural Design• The importance of having an integration “blueprint”
– Information Coverage – Master of Record– Direction of Flow
• Pre-requisites for designing a workable solution– Insights into business process– Familiarity with software– Knowledge of the data
• Integration paradigms– Middleware vs Direct
Monday, April 8, 13
![Page 10: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/10.jpg)
Typical Technical “Blueprint”1. Integration Goals / Summary2. Integration Dependencies3. Integration Method4. Integration Requirements
4.1. Product Information4.1.1. Inventory4.1.2. Pricing
4.1.2.1. General Pricing4.1.2.2. Customer Tiers
4.2. Customer Accounts4.3. Orders
4.3.1. Export4.3.2. Fulfillment / Updates4.3.3. Payment Processing
Monday, April 8, 13
![Page 11: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/11.jpg)
Facing Challenges• Using COTS (Commercial Off the Shelf) products• Controlling data exposure at connection points
– aka PCI-DSS compliance• Real-time data synchronization: Is it possible or even worth it?• Maintaining data integrity between disparate systems• Overall reliability of the integration processes
Monday, April 8, 13
![Page 12: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/12.jpg)
Engineering Solutions
• Methods of Direct Integration• Managing Memory Leaks• Building a Scalable Integration
Monday, April 8, 13
![Page 13: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/13.jpg)
Methods of Direct Integration• Database
– Remote– Local
• Flat Files• Web Services• Combinations
Monday, April 8, 13
![Page 14: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/14.jpg)
Managing Memory Leaks• PHP 5.3 Garbage Collector
– Largely addressed circular reference memory leaks– Far better than PHP 5.2 but doesn’t eliminate all possible leaks– Global anonymous functions can still be orphaned!
• Looping over all records of a large catalog in one execution is a practical impossibility... particularly when running PHP 5.2 or GC disabled
• Using memory_get_usage and a bit of clever logic:– We can prevent processes dying due to memory leaks– Create a robust and reliable process manager
Monday, April 8, 13
![Page 15: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/15.jpg)
CreateCreate owner/repository
Overview Commits Pull requests Downloads
Repositories D
integrating-magento-with-an-erp davidalger Following Share
CloneClone ForkFork CompareCompare Pull requestPull request
Source B
integrating-magento-with-an-erp / CLS_Integration_Model_Process_Abstract.php
1 2 3 4 5 6 7 8 910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061
<?php/** * Abstract.php * * @category CLS * @package Integration * @author David Alger <[email protected]> * @copyright Copyright (c) 2013 David Alger & Classy Llama Studios, LLC */
abstract class CLS_Integration_Model_Process_Abstract{ protected static function _getMemLimit() { static $limit = NULL; if ($limit === NULL) { $value = trim(ini_get('memory_limit')); $code = strtolower($value[strlen($value)-1]); switch ($code) { case 'g': // intentional fall through $value *= 1024; case 'm': // intentional fall through $value *= 1024; case 'k': // intentional fall through $value *= 1024; } $limit = (int)$value; } return $limit; } /** * Reschedules the cron job $interval seconds into the future. * * @param Mage_Cron_Model_Schedule $schedule * @param int $interval */ protected function _rescheduleCron(Mage_Cron_Model_Schedule $pSchedule, $interval = 10) { $timestamp = Mage::getSingleton('core/date')->gmtTimestamp()+$interval; $schedule = Mage::getModel('cron/schedule'); $schedule->setJobCode($pSchedule->getJobCode()) ->setStatus(Mage_Cron_Model_Schedule::STATUS_PENDING) ->setCreatedAt($timestamp) ->setScheduledAt($timestamp) ; $schedule->save();
return $this; }
/** * Takes a data collections and process it within a memory monitoring loop * * @param CLS_Integration_Model_Resource_Erp_Db_Collection_Abstract $collection * @param string $callback */ protected function _processDataCollection($collection, $callback) { $index = 0; $limit = self::_getMemLimit(); // store the memory limit in bytes for calculations $baseline = 0; // the current memory usage at last iteration $delta = 0; // maximum difference in memory usgae from one iteration to the next $space = NULL; // the remaining number of iterations we have based on the $delta
mastermaster SourceSource
DiffDiff
HistoryHistory
5128d535128d53 17 minutes ago17 minutes ago BlameBlame Full commitFull commit
RawRaw
Monday, April 8, 13
![Page 16: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/16.jpg)
CreateCreate owner/repository
Overview Commits Pull requests Downloads
Repositories D
integrating-magento-with-an-erp davidalger Following Share
CloneClone ForkFork CompareCompare Pull requestPull request
Source B
integrating-magento-with-an-erp / _processDataCollection.php
1 2 3 4 5 6 7 8 91011121314151617181920212223242526272829303132333435363738394041424344454647484950515253
<?php/** * Abstract.php * * @category CLS * @package Integration * @author David Alger <[email protected]> * @copyright Copyright (c) 2013 David Alger & Classy Llama Studios, LLC */
abstract class CLS_Integration_Model_Process_Abstract{ /** * Takes a data collections and process it within a memory monitoring loop * * @param CLS_Integration_Model_Resource_Erp_Db_Collection_Abstract $collection * @param string $callback */ protected function _processDataCollection($collection, $callback) { $index = 0; $limit = self::_getMemLimit(); // store the memory limit in bytes for calculations $baseline = 0; // the current memory usage at last iteration $delta = 0; // maximum difference in memory usgae from one iteration to the next $space = NULL; // the remaining number of iterations we have based on the $delta foreach ($collection as $record) { $baseline = memory_get_usage(); // update the baseline try { $this->$callback($record); // process the record } catch (Zend_Db_Exception $e) { // catch, log and skip items where an exception (like a deadlock or lock timeout) occurs... Mage::logException($e); continue; } if ($index == 0) { $baseline = memory_get_usage(); // first iteration, update this post-processing to avoid inflated delta } else { $delta = max($delta, memory_get_usage() - $baseline, 0.0001); // calculate memory usage delta $space = floor(($limit - memory_get_usage()) / $delta); // calculate approximate space for iterations } // if we have space for less than 100 estimated iteration remaining, log a message and break to cleanup if ($space !== NULL && $space <= 100) { Mage::log("CLS_Integration [".__CLASS__."::".__FUNCTION__."]: Must terminate, within 100" ." iterations of remaining space allowed by memory_limit!"); return false; } } return true; }}
mastermaster SourceSource
DiffDiff
HistoryHistory
5f908555f90855 15 seconds ago15 seconds ago BlameBlame Full commitFull commit
RawRaw
Blog · Report a bug · Support · Documentation · API · Forum · Server status · Terms of service · Privacy policy
Monday, April 8, 13
![Page 17: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/17.jpg)
Building a Scalable Integration• Dealing with large amounts of data
– Polling for changes– Data write throughput– Index management
• Managing integration processes– Use the built-in cron dispatcher for job execution– Thread locks are critical to avoid racing and overworking jobs– A page cursor can come in very handy when polling is used
Monday, April 8, 13
![Page 18: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/18.jpg)
CreateCreate owner/repository
Overview Commits Pull requests Downloads
Repositories D
integrating-magento-with-an-erp davidalger Following Share
CloneClone ForkFork CompareCompare Pull requestPull request
Source B
integrating-magento-with-an-erp / CLS_Integration_Model_Process_Abstract.php
1 2 3 4 5 6 7 8 910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061
<?php/** * Abstract.php * * @category CLS * @package Integration * @author David Alger <[email protected]> * @copyright Copyright (c) 2013 David Alger & Classy Llama Studios, LLC */
abstract class CLS_Integration_Model_Process_Abstract{ protected static function _getMemLimit() { static $limit = NULL; if ($limit === NULL) { $value = trim(ini_get('memory_limit')); $code = strtolower($value[strlen($value)-1]); switch ($code) { case 'g': // intentional fall through $value *= 1024; case 'm': // intentional fall through $value *= 1024; case 'k': // intentional fall through $value *= 1024; } $limit = (int)$value; } return $limit; } /** * Reschedules the cron job $interval seconds into the future. * * @param Mage_Cron_Model_Schedule $schedule * @param int $interval */ protected function _rescheduleCron(Mage_Cron_Model_Schedule $pSchedule, $interval = 10) { $timestamp = Mage::getSingleton('core/date')->gmtTimestamp()+$interval; $schedule = Mage::getModel('cron/schedule'); $schedule->setJobCode($pSchedule->getJobCode()) ->setStatus(Mage_Cron_Model_Schedule::STATUS_PENDING) ->setCreatedAt($timestamp) ->setScheduledAt($timestamp) ; $schedule->save();
return $this; }
/** * Takes a data collections and process it within a memory monitoring loop * * @param CLS_Integration_Model_Resource_Erp_Db_Collection_Abstract $collection * @param string $callback */ protected function _processDataCollection($collection, $callback) { $index = 0; $limit = self::_getMemLimit(); // store the memory limit in bytes for calculations $baseline = 0; // the current memory usage at last iteration $delta = 0; // maximum difference in memory usgae from one iteration to the next $space = NULL; // the remaining number of iterations we have based on the $delta
mastermaster SourceSource
DiffDiff
HistoryHistory
5128d535128d53 17 minutes ago17 minutes ago BlameBlame Full commitFull commit
RawRaw
Monday, April 8, 13
![Page 19: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/19.jpg)
CreateCreate owner/repository
Overview Commits Pull requests Downloads
Repositories D
integrating-magento-with-an-erp davidalger Following Share
CloneClone ForkFork CompareCompare Pull requestPull request
Source B
integrating-magento-with-an-erp / cls_integration_lock.sql
1 2 3 4 5 6 7 8 910111213141516171819
-- -- cls_integration_process_lock - table schema--
CREATE TABLE `cls_integration_process_lock` ( `code` varchar(50) NOT NULL COMMENT 'Job Code', `locked_at` timestamp NULL DEFAULT NULL COMMENT 'Locked At', `freed_at` timestamp NULL DEFAULT NULL COMMENT 'Freed At', `status` smallint(6) NOT NULL DEFAULT '0' COMMENT 'Lock Status', PRIMARY KEY (`code`)) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Holds an atomically set lock to prevent overlapping jobs.';
-- -- cls_integration_process_lock - atomic lock update-- ref: CLS_Integration_Model_Resource_Process_Lock::obtainLock--
UPDATE `cls_integration_process_lock` SET `status` = 1 WHERE `status` = 0 AND `code` = 'product';
mastermaster SourceSource
DiffDiff
HistoryHistory
8e7c0768e7c076 a minute agoa minute ago BlameBlame Full commitFull commit
RawRaw
Blog · Report a bug · Support · Documentation · API · Forum · Server status · Terms of service · Privacy policy
English · Git 1.7.10.3 · Mercurial 2.2.2 · Django 1.3.1 · Python 2.7.3 · ec2dce67cf18 / ad77ca6c8d0a @ bitbucket01Monday, April 8, 13
![Page 20: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/20.jpg)
CreateCreate owner/repository
Overview Commits Pull requests Downloads
Repositories D
integrating-magento-with-an-erp davidalger Following Share
CloneClone ForkFork CompareCompare Pull requestPull request
Source B
integrating-magento-with-an-erp / cls_integration_lock.sql
1 2 3 4 5 6 7 8 910111213141516171819
-- -- cls_integration_process_lock - table schema--
CREATE TABLE `cls_integration_process_lock` ( `code` varchar(50) NOT NULL COMMENT 'Job Code', `locked_at` timestamp NULL DEFAULT NULL COMMENT 'Locked At', `freed_at` timestamp NULL DEFAULT NULL COMMENT 'Freed At', `status` smallint(6) NOT NULL DEFAULT '0' COMMENT 'Lock Status', PRIMARY KEY (`code`)) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Holds an atomically set lock to prevent overlapping jobs.';
-- -- cls_integration_process_lock - atomic lock update-- ref: CLS_Integration_Model_Resource_Process_Lock::obtainLock--
UPDATE `cls_integration_process_lock` SET `status` = 1 WHERE `status` = 0 AND `code` = 'product';
mastermaster SourceSource
DiffDiff
HistoryHistory
8e7c0768e7c076 a minute agoa minute ago BlameBlame Full commitFull commit
RawRaw
Blog · Report a bug · Support · Documentation · API · Forum · Server status · Terms of service · Privacy policy
English · Git 1.7.10.3 · Mercurial 2.2.2 · Django 1.3.1 · Python 2.7.3 · ec2dce67cf18 / ad77ca6c8d0a @ bitbucket01Monday, April 8, 13
![Page 21: Happy Together: Creating Successful Magento ERP Integrations | Imagine 2013…](https://reader035.vdocument.in/reader035/viewer/2022081401/5564ca9ed8b42ad9498b4609/html5/thumbnails/21.jpg)
Questions or Comments?
David Alger (@blackbooker)CTO / Lead Engineer
http://bit.ly/imagine-erp
P.S. W
E’RE H
IRIN
G!!!
Monday, April 8, 13