postgresql built your car
TRANSCRIPT
![Page 1: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/1.jpg)
PostgreSQL Built Your Car
Auto Industry Data ExchangeUsing Open Source Solutions
Aaron ThulAutoweb Communications
http://www.autoweb.net
O’Reilly Open Source ConventionAugust 1 5, 2005
![Page 2: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/2.jpg)
Did PostgreSQL Really Build a Car?
• No not really• Engineer and Designer likely used Autoweb• ADXi (Autoweb Data eXchange internet)• AGM (Autoweb Gateway Manager)• Both built on free and open source solutions.
![Page 3: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/3.jpg)
Autoweb and ADX at a Glance
• Presence in Over 38 Countries• Over 20,000 users in 2,500 Companies• Key Clients
– General Motors– Ford Motor Company– DaimlerChrysler– Nissan– Mitsubishi– Honda
![Page 4: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/4.jpg)
With PostgreSQL and other Open Source software Autoweb is
• Transferring thousands of transactions per day• About 1.5 documents per transaction• 75 Gigabytes per day of change• Processing 221 Gigabytes of data • Managing an average of 100 simultaneous users
![Page 5: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/5.jpg)
ADX Data
• Very large engineering documents• Files are stored on a shared NFS server• All ADXi web severs use the standard shared NFS mount• Data moves via an an integrated routing module• All metadata is stored in a database• Each transaction supports multiple recipients
![Page 6: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/6.jpg)
Metadata
• Who Sent the File?• Who should get the File?• File Type?• Conversion Information?• Transaction history?• Did the file get delivered (Transaction Reports)?
![Page 7: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/7.jpg)
Legacy Application
• IBM Servers• AIX• Oracle 8• RSA SecureID• Stronghold Web Server• Compiled C CGI Scripts
![Page 8: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/8.jpg)
Second Generation ADXi
• Lower Cost Comity Servers• Easy Deployment of Additional Servers• Reduced Management of Software Licenses• Reduce Cost to Deploy Additional Boxes• Remember there are other costs (Support) • Improve application availability
![Page 9: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/9.jpg)
ADXi Building Blocks
• Debian Stable (3.1)• PostgreSQL (7.4)• SlonyI (1.0.5)• Apache (1.3)• Perl (5.8)• Mod_perl (1.29)• NetApp and EMC Clarion NAS
![Page 10: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/10.jpg)
AGM (Autoweb Gateway Manager) Building Blocks
• Low Cost x86 Server• Debian Stable (3.1)• PostgreSQL (7.4)• Apache (1.3)• Perl (5.8)• Mod_perl (1.29)• With AGMs Autoweb is pushing these open source solutions into
many enterprise datacenters for the first time.• Each AGM is deployed to a customers data center.
![Page 11: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/11.jpg)
Updates
• We make our own Debian compliant debs• Use our own APT repository for ADX• PostgreSQL Perl and other dependencies are standard Debian
versions• All ADX servers only need to do:
aptget update ; aptget install adx• Wrote custom scrips to handle detect and handle database schema
changes with SlonyI
![Page 12: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/12.jpg)
Why PostgreSQL?
• Evaluated MySQL and PostgreSQL to replace Oracle 8• Required features for replacement database
– Triggers– Foreign Keys– Subselects– Compatible Syntax
![Page 13: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/13.jpg)
Why mod_perl?
• New application started in Perl• Wanted better performance• Needed database connection caching• Majority of overhead is only paid once, first page load on a new
apache processes.
![Page 14: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/14.jpg)
SlonyI
• Relies on triggers in the database• When a failed server is brought back online it will
automatically start catching up • Unfortunately when the master fails, clients need to
reconnect to the new master and likely be reconfigured
![Page 15: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/15.jpg)
PgPool
• Pgpool can detect the failure of a server and failover automatically.
• Nodes can be added and removed for maintenance without effecting PostgreSQL.
• Can only manage 2 servers at most.• Sends all commands to both databases.
![Page 16: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/16.jpg)
SlonyI on the database servers and pgpool local on the web server
• All the features of SlonyI with auto fail over• We monitor for server failure each database server does
inserts into each other database to make sure they are alive.
• If slave can connect to web servers and can not connect to master, it makes promotes it self to master and knocks old master out of VLAN via SNMP.
• Helping to move ADXi to 99.999% availability
![Page 17: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/17.jpg)
What does all this look like
Web Load BalancerP ow e rV au lt
775 N
P ow e rV au lt775 N
P ow e rV au lt775 N
PgSQLSlonyI Slave
P ow e rV au lt775 N
P ow e rV au lt775 N
PgSQLSlonyI Master
EthernetLAN Switches
ADXi App ServersApache
PerlPgpool
Clients
![Page 18: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/18.jpg)
Results
• This architecture has transferred– 522,000 Transactions– 815,000 Files– To 585,000 Receivers
![Page 20: PostgreSQL Built Your Car](https://reader035.vdocument.in/reader035/viewer/2022071602/613d5c8f736caf36b75c6c1c/html5/thumbnails/20.jpg)