the forrester wave™ - oracle · 2016-10-03 · oracle data integrator oracle goldengate oracle...

58

Upload: others

Post on 21-Mar-2020

23 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database
Page 2: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

TheForresterWave™:GeospaEalAnalyEcsToolsAndPlaGorms,Q32016

“WhilehardcoreGISprofessionalsmaystarttheirworkinotherapplicaEons,whentheywanttosolvespaEalproblemsinproducEonandwithweb-andIoT-scaledata,OraclegivesthemtheplaGormtodoso.”

2

Analysts:RowanCurranwithHolgerKisker,Ph.D.andEmilyMillerSeptember1,2016

Geospatial Analytics Tools And Platforms

TheForresterWave™iscopyrightedbyForresterResearch,Inc.ForresterandForresterWave™aretrademarksofForresterResearch,Inc.TheForresterWave™isagraphicalrepresenta>onofForrester'scallonamarketandisplo@edusingadetailedspreadsheetwithexposedscores,weigh>ngs,andcomments.Forresterdoesnotendorseanyvendor,product,orservicedepictedintheForresterWave.Informa>onisbasedonbestavailableresources.Opinionsreflectjudgmentatthe>meandaresubjecttochange.

Page 3: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

BuildApplicaEonsonSpark,Hadoop,andNoSQLwithOracleBigDataSpaEalandGraphJimSteinerVicePresident,ProductManagementSivaRavadaSeniorDirector,ProductDevelopmentZheWuArchitect,ProductDevelopmentSeptember22,2016

Page 4: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirecEon.ItisintendedforinformaEonpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfuncEonality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andEmingofanyfeaturesorfuncEonalitydescribedforOracle’sproductsremainsatthesolediscreEonofOracle.

4

Page 5: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ProgramAgenda

5

BigDataSpaEalandGraphfeatureoverview

CommonUseCasesandCodingExamples

BenefitsfortheDeveloper

QuesEonsandAnswers

1

2

3

4

Page 6: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.| 6

OracleBigDataSpaEalAndGraph

Page 7: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ACompleteBigDataPorGolio

CREATEVALUEFROMDATA

OracleDataIntegratorOracleGoldenGateOracleBigDataPreparaEon

OracleBigDataSQLOracleBigDataConnectorsOracleNoSQLDatabase

OracleBIEE+VisualAnalyzerOracleBigDataDiscoveryOracleBigDataSpaEalandGraphOracleAdvancedAnalyEcs

OracleBigDataApplianceOraclePublicCloudOracleBigDataCloud

OracleData-as-a-Service

OracleApplicaEons

BIGDATAMANAGEMENT

BIGDATAINTEGRATION

BIGDATAANALYTICS

BIGDATAINFRASTRUCTURE

Page 8: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

BigDataDefined

Big Data: Techniques and Technologies that Enable Enterprises to Effectively and Economically Analyze All of their Data

Page 9: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

RequiresmoredevelopmentresourcesanddatascienEsts

BigDataChallengesBuildyourownenvironmentfromcommodityhardwareandopensourcesoiware.

Page 10: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

MakedevelopersanddatascienEstsmoreproducEve

BigDataSoluEonOpEmized,pre-configuredBigDataApplianceandCloudplaGorms•  Popularlanguages,developmentframeworksandserviceAPIs

•  In-Memory,parallelOOTBGraphAlgorithms•  Massively-scalableGraphDatabase•  Pre-built,parallel,MapReduceandSparkSpaEalAlgorithms•  RasterandVectorprocessingFrameworks

BigData

Page 11: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

GraphandSpaEalAnalysis–ItisaboutrelaEonships

•  ArethingsinthesamelocaEon?Whoisthenearest?Whattaxzoneisthisin?Wherecandeliverin35minutes?Whatisinmysalesterritory?Isthisbuiltinafloodzone?

•  WhichsupplieramImostdependentupon?WhoisthemostinfluenEalcustomer?DomyproductsappealtocertaincommuniEes?Whatpamernsarethereinfraudulentbehavior?

Page 12: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

BigDataSpaEalandGraphBigDataoienneedsanOrganizingprinciple–DataHarmonizaEonBigDataanalysisisoienaboutrelaEonshipsnotaggregaEonBigDataplaGormiseconomicallycompellingforworkingwithmassivedatasetsfoundinspaEalandgraphworkflows

Page 13: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

MoEvaEonforOracleBigDataSpaEal

EmergenceofHadoopforspaEalanalysisinbusinessandspaEalworkflowsExisEngHadoop-basedSpaEaltechnologiesareGeoSpaEal-centricnotApplicaEon-centricNosignificantcommercialofferings

Page 14: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

CategorizaEonandfilteringbasedonlocaEonandproximity

PreparaEon,validaEonandcleansingofSpaEalandRasterdata

Visualizinganddisplayingresultsonamap

SpaEalqueryingandanalysisofHadoopdatawithSQL

WhatproblemscanBigDataSpaEalanalysisaddress?

DataHarmonizaEonusinganylocaEonamribute(address,postalcode,lat/long,placename,etc).

Copyright©2016Oracleand/oritsaffiliates.Allrightsreserved.|

Page 15: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

DataenrichmentserviceAPIusingGeoNamesandgeometryhierarchydata

MapReducerouEnesfordistancecalculaEons,PointInPolygon,buffercreaEon,CategorizaEon,KMeansClustering,Binning

SpaEalprocessingofdatastoredinHDFSorNoSQL.RasterprocessingoperaEons:Mosaicandsub-setoperaEons.GeodeEcandCartesiandata

HTML5MapVisualizaEonAPI

HiveSQLAPIQueryfromOracleDBwithBigDataSQL&OracleSQLConnectorsforHadoop

BigDataSpaEalFeatures

Page 16: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

GraphDataModel

Whatisagraph?–  Asetoflinksandnodes(andopEonallyamributes)–  Agraphissimplylinkeddata

Whydowecare?–  Graphsareeverywhere

•  Socialnetworks/SocialWeb(Facebook,Linkedin,Twimer,Baidu,Google+,…)•  Cybernetworks,powergrids,proteininteracEongraphs•  Knowledgegraphs(IBMWatson,AppleSIRI,GoogleKnowledgeGraph)

–  GraphsareintuiEveandflexible•  Easytonavigate,easytoformapath,naturaltovisualize•  Donotrequireapredefinedschema

E

A D

C B

F

Page 17: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.| 17

PropertyGraph:BigDataSpaEalandGraphFastanalyEcswithhorizontallyscalablestorage

•  ParallelIn-memory-basedAnalyst(PGX)–  39built-inmemory-basedgraphanalysisalgorithms–  PropertyGraphQueryLanguage(PGQL)–  Smartfilteringoflargegraphs

•  Flexibleinterfaces–  Python,Groovy–  Java,Tinkerpop,Blueprints,Gremlin–  ApacheLuceneandSolrCloud

•  Massively-ScalableGraphDatabase– MulEpleback-ends:NoSQL,Hbase,OracleDatabase–  Scalessecurelyto10sofbillionsofnodes/edges

ScalableandPersistentStorage

GraphDataAccessLayerAPI

GraphAnalyEcsIn-memoryAnalyEcEngine

RESTWebService

Blueprints&SolrCloud/Lucene

PropertyGraphStorageonApacheHBaseandOracleNoSQL

Python,Perl,PHP,Ruby,Javascript,…

JavaAPIs

JavaAPIs

Page 18: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

Richsetofbuilt-inparallelgraphalgorithms …andparallelgraphmutaEonoperaEons

ComputaEonalAnalyEcs:Built-inPackage

DetecUngComponentsandCommuniUes

Tarjan’s,Kosaraju’s,WeaklyConnectedComponents,LabelPropagaEon(w/variants),SomanandNarang’sSpacificaEon

RankingandWalking

Pagerank,PersonalizedPagerank,BetwennessCentrality(w/variants),ClosenessCentrality,DegreeCentrality,EigenvectorCentrality,HITS,Randomwalkingandsampling(w/variants)

EvaluaUngCommunityStructures

∑ ∑

Conductance,ModularityClusteringCoefficient(TriangleCounEng)Adamic-Adar

Path-Finding

Hop-Distance(BFS)Dijkstra’s,Bi-direcEonalDijkstra’sBellman-Ford’s

LinkPredicUon SALSA(Twimer’sWho-to-follow)

OtherClassics VertexCoverMinimumSpanning-Tree(Prim’s)

a

d

b eg

c i

f

h

Theoriginalgrapha

d

b eg

c i

f

h

CreateUndirectedGraph

SimplifyGraph

a

d

b eg

c i

f

h

LeiSet:“a,b,e”a d

b

e

g

c

i

CreateBiparUteGraph

ge b d i a f c h

Sort-By-Degree(Renumbering)

FilteredSubgraph

d

bg

i

e

18

Page 19: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ModelingandAnalyzingTheInternetofThingsMassivescale–TrillionsofConnecEonsDynamicNetworkIncludesmetadataaswellasevents/enricheddataExtensiblebyotherdatasource

19

Page 20: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

OnPremiseorCloud

BigDataCloud

20

Page 21: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

•  Fast,reliableandsecureserviceonOracleBigDataAppliance

•  FullyautomatedserviceforHadoopandSpark

•  IncludesBigDataSpaEalandGraphinsubscripEon

OracleBigDataCloudService

Page 22: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ProgramAgenda

22

BigDataSpaEalandGraphfeatureoverview

CommonUseCasesandCodingExamples

BenefitsfortheDeveloper

QuesEonsandAnswers

1

2

3

4

Page 23: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.| 23

hmp://semanEcs.us.oracle.com:8988/viewlets/pg_2016_training/oow2016/oow2016_pg_2016_tax_fraud_with_cycle_detecEon_and_visualizaEon/oow2016_pg_2016_tax_fraud_with_cycle_detecEon_and_visualizaEon_viewlet_swf.html

Java,Groovy,BDSG,OracleNoSQLDatabase,PGQL,CytoscapeVisualizaUon,GraphAnalyEcs

Page 24: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.| 24

Java,Groovy,BDSG,ApacheHBase,Solr,GraphAnalyUcs,GraphNavigaUon,andPython

hmp://semanEcs.us.oracle.com:8988/viewlets/pg_2016_training/oow2016/oow2016_pg_2016_training_recommendaEon_ppr_with_python_chart/oow2016_pg_2016_training_recommendaEon_ppr_viewlet_swf.html

Page 25: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

BuildingSpaEalAnalyEcalApplicaEonswithJavaandPython

25

Page 26: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

RasterDemoMovie

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 26

Page 27: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ImageProcessingwithfeaturehighlitghing• GivenasetofrasterdataforelevaEon,calculatetheHillShadefromtherasterdata

•  Input:SinglebanddigitalelevaEonmodeldata

• Output:Singlebandrasterimageshowingthehillshade

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 27

Page 28: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

• Input:NXMpixelswhereeachpixelisafloaEngpointnumberdenoEngelevaEon• FindtheshadedrelieffromtheDEM• Algorithm

• Lookatthevaluesof8neighborsandthecurrentpixelvalueandgenerateanewpixel• Needstheneighbor’spixelvaluetocalculatethenewpixelvalueforthecurrentpixel

ShadedReliefcalculaEon

Page 29: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

RasterDataAnalysisFramework•  TradiEonalalgorithmwillworkverywellinMap-Reduceframework• OncethedataisloadedintoHDFSrestoftheprocessingcanbedoneusingstandardalgorithms

• VeryeffecEveforrasterdataprocessingasmanymap-reducenodescanworktogethertoproducetheresultinashortamountofEme

• Customersdon’tneedtowritemap-reducealgorithmsforimageprocessing• AlltheoperaEonsareperformedonacatalogueofimages

– Customerscanlogicallycombineanynumberofimagesintoacatalogueorintoavirtualmosaic

Page 30: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

RasterDataprocessingworkflow• CopytheimagefilesfromNFStoHDFS

– UsetheprovidedETLtools,noneedtowritecodeforthis• ProcesstheimageswithMap-Reduce

– Weprovidetheframeworktoconvertsinglenodeimageprocessingclassestomap-reduce

– Usersneedtoprovidetheimageprocessingclassforanycustomimageprocessingandalanysis

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 30

Page 31: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

CopytheinputRasterdatatoHDFS

#********Executethescriptwithavalidyarnuserwithaccess************

if__name__=="__main__":

iflen(sys.argv)!=3:

print"usage:imageloader.py<innputLocalFolder><relaEveHDFSPath>where\n"\

"<innputLocalFolder>isacommaseparatedstringwithmulEplefilesorfolderswithimages.\n"\

"<relaEveHDFSPath>HDFSrelaEvepathoftheuserexecuEngthejobtowritetheoutput.\n"

else:

try: printConf()

doLoader(sys.argv[1],sys.argv[2])

listResult(sys.argv[2])

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 31

UsingPYTHONscript

Page 32: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

CopytheinputRasterdatatoHDFS

#*************************************************

#TO-DOConfiguretheseproperEesasnecessary

#************************************************

SHARED_DIR ="/net/host100/scratch/abcduser/test/"

INSTALLATION ="/opt/oracle/oracle-spaEal-graph/"

ALL_ACCESS_DIR =SHARED_DIR+"write/"

GDAL_LIB =ALL_ACCESS_DIR+"gdal"

GDAL_DATA =SHARED_DIR+"data"

RASTER_INST =INSTALLATION+"spaEal/raster/"

LOADER_JAR =RASTER_INST+"jlib/"+"hadoop-imageloader.jar"

#Addmorehadoop_classpathjarsasnecessary

CLASSPATH_JARS =["hadoop-imageprocessor.jar","hadoop-raster-fwk-api.jar","gdal.jar"]

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 32

UsingPYTHONscripts

Page 33: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

CalculatetheslopeprivatedoublecalculateSlope(double[]neighboors,doublepixelWidthX,doublepixelWidthY){

doubleslope=0;

dx=((neighboors[0]+(2*neighboors[3])+neighboors[6])-

(neighboors[2]+(2*neighboors[5])+neighboors[8]))/(8*pixelWidthX);

dy=((neighboors[6]+(2*neighboors[7])+neighboors[8])-

(neighboors[0]+(2*neighboors[1])+neighboors[2]))/(8*pixelWidthY);

slope=Math.sqrt(Math.pow(dx,2)+Math.pow(dy,2));

returnslope;

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 33

Page 34: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

CalculateHillshadedoublezenithRadians=90-alEtude*Math.PI/180;doubleazimuthRadians=azimuth_math*Math.PI/180;clearNeighborhood(neighborhood);slopeRadians=Math.atan(slope*zFactor);aspectRadians=calculateAspectRadians();hillShade=calculateHillShade(zenithRadians,azimuthRadians,slopeRadians,aspectRadians);

finalFloatData[posiEon]=hillShade;

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 34

Page 35: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ExecuEngtheanalysisprocessasamap-reducejobNowthatthecatalogandmosaicconfiguraEonareset,executethejobusingthehadoopjarcommandwiththehadoop-imageprocessor.jarandalltheopEonsdescribedinthelaststepsandwaitunElitfinishes.

hadoopjar/opt/oracle/oracle-spaEal-graph/spaEal/raster/jlib/hadoop-imageprocessor.jar-catalog/opt/oracle/oracle-spaEal-graph/spaEal/raster/Raster-HOL/data/xml/inputDEM.xml-config/opt/oracle/oracle-spaEal-graph/spaEal/raster/Raster-HOL/data/xml/testSlopeFS.xml-gdal/opt/oracle/oracle-spaEal-graph/spaEal/raster/gdal/lib-gdalData/opt/oracle/oracle-spaEal-graph/spaEal/raster/gdal/data-usrlib/opt/oracle/oracle-spaEal-graph/spaEal/raster/jlib/external-analysis.jar

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 35

Page 36: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

• DatacontaininglocaEonreferencesisprocessedandcategorizedbasedonacustomgeometryhierarchy

•  Input:TweetdataforJanuary2016Emeperiod

• Output:categorizeddatathatassociateseachtweettoageometryinthehierarchy

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 36

Categorizedatabasedonahierarchy--Java

Page 37: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

LoadtheinputdataonHDFSforspaEalprocessing• AssumethatthedataisstoredonanNFSfilesystemthatisalsoaccessiblefromtheHDFSserver

•  TheinputdataisstoredasatextfileintheJSONformatwithonerecordforonetweethadoopfs-put/opt/oracle/oracle-spaUal-graph/spaUal/vector/HOL/data/tweets.json

/user/oracle/HOL/tweets.json

•  ItisassimpleascopyingthedatafromtheNFStoHDFSusinghaddopfilesystemcommands– Nopre-processingrequiredsothedataisreadytogooncethecopyisdone

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 37

Page 38: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

•  Ifthedataisinstandardknownformatsusersdon’tneedtowriteanycodeforreadingtherecordsandextracEnggeometryfromit

•  Standardformatssupported– JSON,ShapeFiles

•  Forothers,usersneedtoprovidecustomcustomRecordInfoReaderclass– WewillshowanexampleofthisusingJSONformat

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 38

HowtoreadTweetdatatoextractgeometryfromit

Page 39: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

RecordInfoProviderExample

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 39

Page 40: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

SetupthecustomhierarchyusingJSONfiles

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 40

Page 41: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ExampleJSONdataforcustomhierarchy

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 41

Exampleofrecordinthefileeurozone_countries.json:{"type":"Feature","_id":"FRA","geometry":{"type":"Polygon","coordinates":[[1.44136,42.60366,1.47851,42.65168,…,1.44136,42.60366]]},"properEes":{"ConEnent":"EU","Name":"France","Alt_Region":"EMEA","CountryCode":"FRA"},"label_box":[-1.12061,45.13915,6.02255,49.19591]}AndexampleofarecordrepresenEngaFranceprovinceineurozone_provinces.json:{"type":"Feature","_id":"3023519","geometry":{"type":"Polygon","coordinates":[[9.19977,41.36465,9.25876,…,41.36465]]},"properEes":{"Country":"France","ISO":"FRA","StateProvinceName":"Corse","CountryCode_StateProvinceName":"FRA_Corse","CountryName_StateProvinceName":"France_Corse"},"label_box":[8.70974,42.08453,9.53075,42.54732]}

Page 42: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

InvoketheJavaclassestorunthecategorizaEonjobhadoopjar$API_LIB_DIR/sdohadoop-vector.jaroracle.spaEal.hadoop.vector.mapred.job.CategorizaEon-libjars$HADOOP_LIB_JARSspaEalOperaEon=IsInsideinput=/user/oracle/HOL/tweets.jsonoutput=/user/oracle/HOL/euroZoneCatOutputinputFormat=oracle.spaEal.hadoop.vector.geojson.mapred.GeoJsonInputFormatrecordInfoProvider=oracle.spaEal.hadoop.vector.geojson.GeoJsonRecordInfoProvidersrid=8307geodeEc=truetolerance=0.5hierarchyInfo=hol.EuroHierarchyInfohierarchyIndex=/user/oracle/HOL/hierarchyIndexhierarchyDataPaths=file:///opt/oracle/oracle-spaEal-graph/spaEal/vector/HOL/data/eurozone_countries.json,file:///opt/oracle/oracle-spaEal-graph/spaEal/vector/HOL/data/eurozone_provinces.json

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 42

Page 43: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

SimpleHTML5applicaEontoshowtheresultsonamapmap.addLayer(EleLayer);varcs=newOM.style.Color({strokeThickness:1,stroke:"#009400",fill:"#5c85ad",fillOpacity:0.75});varlayer=newOM.layer.VectorLayer("datapack_layer",{def:{type:OM.layer.VectorLayer.TYPE_DATAPACK,url:"saved_results.json",labelColumn:"county"},renderingStyle:cs});

layer.setLabelsVisible(true);layer.setBoundingTheme(true);

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 43

Page 44: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ImageProcessingwithfeatureExtracEon--Python• GivenasetofrasterdataforelevaEon,calculatetheHillShadefromtherasterdata

•  Input:SinglebanddigitalelevaEonmodeldata

• Output:Singlebandrasterimageshowingthehillshade

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 44

Page 45: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

CopytheinputRasterdatatoHDFS

#********Executethescriptwithavalidyarnuserwithaccess************

if__name__=="__main__":

iflen(sys.argv)!=3:print"usage:imageloader.py<innputLocalFolder><relaEveHDFSPath>where\n"\

"<innputLocalFolder>isacommaseparatedstringwithmulEplefilesorfolderswithimages.\n"\

"<relaEveHDFSPath>HDFSrelaEvepathoftheuserexecuEngthejobtowritetheoutput.\n"

else:try: printConf()

doLoader(sys.argv[1],sys.argv[2])

listResult(sys.argv[2])

exceptExcepEonasinst:

printinst.args

printinst

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 45

UsingPYTHONscripts

Page 46: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

CopytheinputRasterdatatoHDFS#*************************************************

#TO-DOConfiguretheseproperEesasnecessary

#************************************************

SHARED_DIR ="/net/host100/scratch/abcduser/test/"

INSTALLATION ="/opt/oracle/oracle-spaEal-graph/"

ALL_ACCESS_DIR =SHARED_DIR+"write/"

GDAL_LIB =ALL_ACCESS_DIR+"gdal"

GDAL_DATA =SHARED_DIR+"data"

RASTER_INST =INSTALLATION+"spaEal/raster/"

LOADER_JAR =RASTER_INST+"jlib/"+"hadoop-imageloader.jar"

OVERLAP ="10"

#Addmorehadoop_classpathjarsasnecessary

CLASSPATH_JARS =["hadoop-imageprocessor.jar","hadoop-raster-fwk-api.jar","gdal.jar"]

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 46

UsingPYTHONscripts

Page 47: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

CalculatetheslopeprivatedoublecalculateSlope(double[]neighboors,doublepixelWidthX,doublepixelWidthY){

doubleslope=0;

dx=((neighboors[0]+(2*neighboors[3])+neighboors[6])-

(neighboors[2]+(2*neighboors[5])+neighboors[8]))/(8*pixelWidthX);

dy=((neighboors[6]+(2*neighboors[7])+neighboors[8])-

(neighboors[0]+(2*neighboors[1])+neighboors[2]))/(8*pixelWidthY);

slope=Math.sqrt(Math.pow(dx,2)+Math.pow(dy,2));

returnslope;

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 47

Page 48: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

CalculateHillshadedoublezenithRadians=90-alEtude*Math.PI/180;doubleazimuthRadians=azimuth_math*Math.PI/180;clearNeighborhood(neighborhood);slopeRadians=Math.atan(slope*zFactor);aspectRadians=calculateAspectRadians();hillShade=calculateHillShade(zenithRadians,azimuthRadians,slopeRadians,aspectRadians);

finalFloatData[posiEon]=hillShade;

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 48

Page 49: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ExecuEngtheanalysisprocessasamap-reducejobNowthatthecatalogandmosaicconfiguraEonareset,executethejobusingthehadoopjarcommandwiththehadoop-imageprocessor.jarandalltheopEonsdescribedinthelaststepsandwaitunElitfinishes.

hadoopjar/opt/oracle/oracle-spaEal-graph/spaEal/raster/jlib/hadoop-imageprocessor.jar-catalog/opt/oracle/oracle-spaEal-graph/spaEal/raster/Raster-HOL/data/xml/inputDEM.xml-config/opt/oracle/oracle-spaEal-graph/spaEal/raster/Raster-HOL/data/xml/testSlopeFS.xml-gdal/opt/oracle/oracle-spaEal-graph/spaEal/raster/gdal/lib-gdalData/opt/oracle/oracle-spaEal-graph/spaEal/raster/gdal/data-usrlib/opt/oracle/oracle-spaEal-graph/spaEal/raster/jlib/external-analysis.jar

ConfidenEal–OracleInternal/Restricted/HighlyRestricted 49

Page 50: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

DifferenEators--SpaEal• Commercial,supportedsoiware• BigData/MapReduceapproachvs.GIS-relaEonalapproach

– WorksonanydatathatincludeslocaEoninfo,datatypeandfileformat

• GeoEnrichmentservicesincludingglobalgeographichierarchy• Veryrichset(~50)ofspaEaloperatorsandfuncEons•  SupportsbothspaEalprocessingandspaEalanalyEcs• BothVectorandRasterservices•  IncludesmapvisualizaEon

Page 51: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

DifferenEators--Graph• Commercial,supportedsoiware•  “BestofBothWorlds”GraphDB

– In-memorygraphanalysisalgorithms– Distributedgraphdatabasemodel

•  SQL-likeParallelGraphQueryLanguage(PGQL)• Dozensofpre-builtin-memorygraphanalysisalgorithms• 10-50xfasteranalyEcsthancompeEtors’offerings

• Analyze20-30BillionedgegraphinmemoryonasingleBDAnode

Page 52: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database
Page 53: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

SpaEalandGraphatOpenWorld2016SubUtle

OracleSpaEalandGraphProductTeam

ConfidenEal–OracleInternal/Restricted/HighlyRestricted

Page 54: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

The Spatial & Graph SIG User Community The SIG connects and exchanges knowledge via online communities and at conferences and events

• Chat & Coffee at OOW Join us & chat with other S&G users about your experiences! Bring your own coffee J Wednesday at 8:00 am Meet in front of Oracle Bookstore – Moscone South Lobby

•  Join us online tinyurl.com/oraclespatialcommunity LinkedIn , Google+ & IOUG SIG groups @oraspatialsig | [email protected]

Page 55: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

UpcomingEvents• OracleSpaEalSummitatBIWA2017OracleHQ,RedwoodShores,CAJan.31-Feb.2,2017www.biwasummit.org*Callforspeakersnowopen!

55

CallforSpeakersNow

Open!

Visitwww.biwasummit.org

Page 56: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

LearnMoreAboutOracle’sSpaUalandGraphTechnologies• www.oracle.com/technetwork/database/opEons/spaEalandgraphwww.oracle.com/database/big-data-spaEal-and-graph

• blogs.oracle.comèoraclespaEalèoracle_maps_blogèbigdataspaEalgraph

CVCSpaEalUpdateforDWR

Page 57: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

MoreResources

Page 58: The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ResourcesonBigDataSpaEalandGraph• OracleBigDataSpaEalandGraphonOracle.com:www.oracle.com/database/big-data-spaEal-and-graph

• OTNproductpage(whitepapers,soiwaredownloads,documentaEon,tutorials):www.oracle.com/technetwork/database/database-technologies/bigdata-spaEalandgraph

• OracleBigDataLiteVirtualMachine-afreesandboxtogetstarted:www.oracle.com/technetwork/database/bigdata-appliance/oracle-bigdatalite-2104726.html

• HandsOnLabforBigDataSpaEal:Enyurl.com/BDSG-HOL• Blog–examples,Eps&tricks:blogs.oracle.com/bigdataspaEalgraph

•  @OracleBigData,@SpaEalHannes,@JeanIhm

58