how to migrate data from mongodb to postgres with torodb · who we are how to migrate data from...
TRANSCRIPT
HowtomigratedatafromMongoDBtoPostgreswithToroDB
Whoweare
30/06/2017 Page2HowtomigratedatafromMongoDBtoPostgreswithToroDB
ExpertsAtYourService>Over50specialistsinITinfrastructure> Certified,experienced,passionate
BasedInSwitzerland> 100%self-financedSwisscompany>OverCHF8mio.Turnover
LeadingInInfrastructureServices>Morethan150customersinCH,D&F>Over50SLAsdbiFlexService contracted
dbi services is hiring ([email protected])
Aboutme
30/06/2017 Page3HowtomigratedatafromMongoDBtoPostgreswithToroDB
MehdiBadaConsultant
+41799287548
mehdi.bada[at]dbi-services.com
Agenda
1.Introduction
2.MongoDB
3.ToroDB
4.Migration: from MongoDB to PostgreSQL
5.Conclusion
30/06/2017 Page4HowtomigratedatafromMongoDBtoPostgreswithToroDB
30/06/2017 Page5HowtomigratedatafromMongoDBtoPostgreswithToroDB
1
2
3
4
5
Introduction
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page6
Introduction
30/06/2017 Page7HowtomigratedatafromMongoDBtoPostgreswithToroDB
1
2
3
4
5
MongoDB
> Overview> DataModel> Highavailability> Horizontalscalability> Limitations
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page8
MongoDBOverview
OpenSource
Horizontalscalability
Schema-less
Documentoriented
Highavailability
Indexes
Richqueries
FullCRUDsupport
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page9
MongoDBOverview
Conceptmapping
RDBMS MongoDB
Tables Collections
Rows/records Documents
Queries returnrows Queriesreturncursor
Join Embedded document
Partition Shard
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page10
MongoDBDataModel
Dataarestoredasdocuments
MongoDBstoresBSONdocuments(BinaryJSON)
Analogoustoadatabaserow
KeysandValues> Key:String> Valuestypes:
> String,number,Boolean,null,array,object
Specialkey:_id> Uniqueidentifier> Objectid:
{“_id”:ObjectId(“56e92b9cfdf7bc92bbb3b51f”),“first_name”:“Mike”,“surname”:“Brody”,“city”:”New-York”,“year”:1987,
}
Date MAC@ PID Counter
12byteHexString
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page11
MongoDBDataModel
Relational MongoDBschema
{“first_name” :“Durand”,“surname” :“Mike",“city” :“Geneva",“country” :“Switzerland”,“cars”:[{“model”:“Ferrari”,“year”:2013}
}
Pers_Id Surname First_name City
0 Mike Durand Geneva
1 Pat Millner London
2 Ortega Alvaro New-York
Car_Id Model Year Pers_Id
0 Ferrari 2013 0
1 Peugeot 2005 0
2 BMW 2016 1
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page12
MongoDBHighavailability
Whichmechanismensurehighavailabilityofyourdata?>MongoDBReplication
HAisachievedthroughautomaticfailover
MongoDBreplicationallows:> Highavailability(HA)
> DisasterRecovery(DR)> Dataduplicationacrossmultipledatabaseservers/storages
> FunctionalSegregation> Topologyofreplicasetscanbeusedfor
> Backups,Analytics,Reporting,DR,Readoperations…
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page13
MongoDBHighavailability
Replicationarchitecture
Application
mongodSecondary
mongodSecondary
mongodPrimary
driver
Replication
Reads Writes
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page14
MongoDBHighavailability
Automaticfailoverprocess mongodPrimary
mongodSecondary
mongodSecondary
Electionofanewprimary
Heartbeat
mongodSecondary
mongodPrimary
Heartbeat
Replication
Newprimaryiselected
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page15
MongoDBHorizontalscalability
Verticalscalability
IncreasingCPU,RAM,I/O
ScalingwithMongoDB>MongoDBSharding
mongodConfigSvr
Shard0
…
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page16
MongoDBHorizontalscalability
ShardingArchitectureApplication
mongodPrimary
mongodSecondary
mongos(queryrouter)
Shard1
…
mongodPrimary
mongodSecondary
mongodConfigSvr
mongodConfigSvr
Sharding
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page17
MongoDBLimitations
NoACIDtransaction> Atomictransactionsonlyworkwithinthesamedocument
[...{user:567,products: [
{id:47,units:7,
},...
]
[...{user:567,orders: [
...{
orderId:24658,products: [
...]
}...]
}...]
CART ORDERS
ThisoperationisnotAtomic!!
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page18
MongoDBLimitations
MongoDBHighavailabilityisnotsafe!!> Datalossdependingtheconsistencylevelyouchoose
MongoDBconsistencylevels>Unacknowledged:Unsafe- 42%ofdataloss> Acknowledged:Unsafe> Journaled:Unsafe> Fsynced:Unsafe> ReplicaAcknowledged:Unsafe>Onlymajorityissafe
https://aphyr.com/posts/322-jepsen-mongodb-stale-reads
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page19
MongoDBLimitations
BIqueryperformancesissues
MongoDBaggregationframeworkisveryslow>Needtoscanmultipledocuments> LotsofI/Orequiredtoanswerthequery
Aggregationonarelationaldesignis“100xfaster”
SolutionforMongoDBBIqueries?> Implementarelationalschema!!
30/06/2017 Page20HowtomigratedatafromMongoDBtoPostgreswithToroDB
1
2
3
4
5
ToroDB
> WhatisToroDB?> Howitworks?> WhyToroDB?
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page21
ToroDBWhatisToroDB?
ThefirstdatabasethatmergesthescalabilityofaNoSQLwiththereliabilityofSQL
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page22
ToroDBWhatisToroDB?
Opensource,document-oriented,JSONdatabasethatrunsontopofPostgreSQL
BIconnectorforMongoDB
JSONdocumentsarestoredrelationally> Significantstorage> I/Osavings
MongoDBdataispersistingintablesandrowswithinaSQLdatabase
Protocols
Abstract Doc
Data2Relational
Backends
MongoDB,Couchbase,Cassandra...
TransformingDocsintoTablesandRows
PostgreSQL,DB2,Oracle,Greenplum...
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page23
ToroDBHowitworks?
ToroDBtransformsdocumentstorelationaltables> Dataisstoredintables> ToroDBanalyzeseveryincomingdocumentandseparatesmetadata(schema)formdata(tuples)> 1+tablesperMongoDBcollections> ToroDBcreatesaRDBMScatalogschemaperMongoDBdatabase> Dynamicandimplicitschemageneration
FullcompatibilitywithMongoDB> APIprograms,clients> CRUDoperationsincludingUPDATE
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page24
ToroDBHowitworks?
ToroDBcanworkasasecondarynodeonaMongoDBreplicaset
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page25
ToroDBWhyToroDB?
NativeSQLBIConnector
Apps:WritedatawithMongoAPI,querywithSQL!
DataIntegrationPlatform:SQLandNoSQLappsinthesameRDBMS
30/06/2017 Page26HowtomigratedatafromMongoDBtoPostgreswithToroDB
1
2
3
4
5
Migration:fromMongoDBtoPostgreSQL
> Overview> Prerequisites> Configuration> Demo
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page27
Migration:fromMongoDBtoPostgreSQLOverview
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page28
Migration:fromMongoDBtoPostgreSQLOverview
ToroDBStampededusesMongoDBreplicasetoplog tokeeptrackofthemodificationsinMongoDB
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page29
Migration:fromMongoDBtoPostgreSQLOverview
Duringthereplication ToroDBStampededtransformsJSONdocumentsintoarelationalschema
ColumnTypeEncoding
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page30
Migration:fromMongoDBtoPostgreSQLPrerequisites
Runtimedependencies
Technology Description
MongoDB InstallandstartMongoDBinstances withthereplication features(dbiservicesbestpracticesinstallation)
Replicasetsconfiguration ToroDBStampedereceivesdatafromaMongoDBreplicaset.Asingle-nodereplicasetissufficient.
PostgreSQL InstallandstartaPostgreSQL instance.Createadedicateduseranddatabase.(dbiservicesbestpractices)
Java ToroDBStampedeiswritteninJavasoaJavaRuntimeEnvironment(JRE)requiredtorunit.Java8isrecommended.
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page31
Migration:fromMongoDBtoPostgreSQLConfiguration
Architecture
MongoEnv PostgresEnv
ToroDBStampededschemageneration
PostgreSQLinstancewithDMKinstalled
MongoDBinstanceswithDMKinstalled
192.168.56.102 192.168.56.140
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page32
Migration:fromMongoDBtoPostgreSQLConfiguration
Afterinstallation,export$TOROHOME variable
CreateandadapttheToroDBconfigurationfile(YAML)
mehdi@MacBook-Pro:/u00/app/torodb/ export TOROHOME /u00/app/torodb/torodb-stampede-1.0.0-beta2”
mehdi@MacBook-Pro:/u00/app/torodb/ echo $TOROHOME/u00/app/torodb/torodb-stampede-1.0.0-beta2
mehdi@MacBook-Pro:/u00/app/torodb/ /u00/app/torodb/torodb-stampede-1.0.0-beta2/bin/torodb-stampede -l > ../conf/torodb.yaml
mehdi@MacBook-Pro:/u00/app/torodb/ vi ../conf/torodb.yaml
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page33
Migration:fromMongoDBtoPostgreSQLConfiguration
ToroDBStampededreadsdatabasescredentialsfromthe.toropassfile
Createthe.toropassfileinthehomedirectory
mehdi@MacBook-Pro:/u00/app/torodb/ echo ”<host>:<port>:<database>:<user>:<PASSWD>" > "$HOME/.toropass"
mehdi@MacBook-Pro:/u00/app/torodb/ chmod 0400 ”~/.toropass”
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page34
Migration:fromMongoDBtoPostgreSQLConfiguration
CustomconfigurationforToroDBStampededonstartup> Commandlineoptions> Configurationfile
Recommendedtouseaconfigurationfile
Printthecurrentconfiguration(YAML)
mehdi@MacBook-Pro:/u00/app/torodb/ ./torodb-stampede -c myconfiguration.yml
mehdi@MacBook-Pro:/u00/app/torodb/ ./torodb-stampede -l
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page35
Migration:fromMongoDBtoPostgreSQLDemo
Steps Description
1 Java8 (JRE)InstallationandConfiguration:http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jre-8u131-linux-x64.tar.gz
2 PostgreSQL InstallationandConfiguration:- Modify,ifneeded,PostgreSQLinstanceconfiguration:/u02/pgdata/PG1/postgresql.conf
https://www.torodb.com/stampede/docs/1.0.0-beta2/configuration/postgresql-configuration-tips/- Createusertorodbanddatabasetorod(withpassword)- Testtheconnectionwithnewuseranddatabases- Adaptpg_hba.conf fornewconnections
3 MongoDBInstallationandConfiguration:- ConfigureandinitializeMongoDBreplication:replset=torodb- Checkreplicationconfig.(primaryandsecondary)- ImportdataintoMongoDB
4 ToroDBStampedeInstallationandConfiguration:- torodb.yaml configurationfilecreation- .toropass filecreationforcreation- StartToroDBStampeded
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page36
Migration:fromMongoDBtoPostgreSQLDemo
30/06/2017 Page37HowtomigratedatafromMongoDBtoPostgreswithToroDB
1
2
3
4
5
Conclusion
> AdvantagesvsDrawbacks
30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page38
ConclusionAdvantagesvsDrawbacks
FastandPowerful
Dynamicschemageneration
OnlysupportsPostgreSQLasRDBMSbackend
Dynamicchangesintheschema
Opensourceandfree!!
Cross-platform:Linux,Windows
Needimprovementsfordifferentschemainthesamemongocollection
Anyquestions?
Pleasedoask!WewouldlovetoboostyourIT-Infrastructure
Howaboutyou?
ZürichBasel
Delémont
Nyon
30/06/2017 Page39HowtomigratedatafromMongoDBtoPostgreswithToroDB