apache nifi: ingesting enterprise data at scale
TRANSCRIPT
![Page 1: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/1.jpg)
1 ©HortonworksInc.2011– 2017.AllRightsReserved
TimothySpann2017FutureofData– PrincetonMeetupHostedbyTRACIntermodal
Apache NiFi: Ingesting Enterprise Data @ Scale
![Page 2: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/2.jpg)
DATAWORKSSUMMIT/HADOOPSUMMITJUNE13–15,2017SanJoseMcHenryConventionCenter
REGISTERNOWANDSAVE$1,000
REGISTERNOW!>
dataworkssummit.com
![Page 3: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/3.jpg)
3 ©HortonworksInc.2011– 2017.AllRightsReserved
Agenda
• Apache NiFi RDBMS, EDI, JSON, CSV, Sensors
• EDI• https://community.hortonworks.com/content/kbentry/59975/in
gesting-edi-into-hdfs-using-hdf-20.html• https://github.com/tspannhw/EnterpriseNIFI
![Page 4: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/4.jpg)
4 ©HortonworksInc.2011– 2017.AllRightsReserved
![Page 5: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/5.jpg)
5 ©HortonworksInc.2011– 2017.AllRightsReserved
![Page 6: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/6.jpg)
6 ©HortonworksInc.2011– 2017.AllRightsReserved
FlowManagement Flowmanagement+StreamProcessing
D A T A I N M O T I O N D A T A A T R E S T
IoTDataSources AWSAzure
GoogleCloudHadoop
NiFiKafka
Storm
Others…NiFi
NiFi NiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
NiFi
HDF2.1– DatainMotionPlatform
EnterpriseServices
Ambari Ranger Otherservices
![Page 7: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/7.jpg)
7 ©HortonworksInc.2011– 2017.AllRightsReserved
Actionable Insights Architecture
IngestionSimpleEventProcessing
EngineComplexEventProcessing
Destination
DataBus
BuildPredictiveModel
FromHistoricalData
DeployPredictiveModel
ForReal-timeInsights
PerishableInsights
HistoricalInsights
![Page 8: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/8.jpg)
8 ©HortonworksInc.2011– 2017.AllRightsReserved
ActionableIntelligenceTransformsIndustrial,Transportation&Utilities
AssetData
CustomerSurveys
Weather&Environmental
ServiceFleetGPSData
SmartMeterStreams
CommodityPrices
REVENUEPROTECTION
SINGLEVIEWOFCUSTOMER
PREDICTIVEEQUIPMENTMAINTENANCE
CONSERVATIONVOLTAGEREDUCTION
COMMODITYTRADING
SocialMedia
GISData
SCADA OutageHistories
CISRecords
EDW
![Page 9: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/9.jpg)
9 ©HortonworksInc.2011– 2017.AllRightsReserved
What is Apache NiFi?
• Created to address the challenges of global enterprise dataflow• Key features:
– VisualCommandandControl
– DataLineage(Provenance)
– DataPrioritization
– DataBuffering/Back-Pressure
– ControlLatencyvs.Throughput
– SecureControlPlane/DataPlane
– ScaleOutClustering
– Extensibility
![Page 10: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/10.jpg)
10 ©HortonworksInc.2011– 2017.AllRightsReserved
Apache NiFi
What is Apache NiFi used for?• Reliable and secure transfer of data between systems• Delivery of data from sources to analytic platforms• Enrichment and preparation of data:
– Conversionbetweenformats– Extraction/Parsing– Routingdecisions
What is Apache NiFi NOT used for?• Distributed Computation• Complex Event Processing• Complex Rolling Window Operations
![Page 11: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/11.jpg)
11 ©HortonworksInc.2011– 2017.AllRightsReserved
NiFi Terminology
FlowFile• Unitofdatamovingthroughthesystem• Content+Attributes(key/valuepairs)
Processor• Performsthework,canaccessFlowFiles
Connection• Linksbetweenprocessors• Queuesthatcanbedynamicallyprioritized
![Page 12: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/12.jpg)
12 ©HortonworksInc.2011– 2017.AllRightsReserved
Contact:
[email protected]/futureofdata-princeton
community.hortonworks.com/users/9304/tspann.html
![Page 13: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/13.jpg)
13 ©HortonworksInc.2011– 2017.AllRightsReserved
HortonworksCommunityConnection
Read access for everyone, join to participate and be recognized
• FullQ&APlatform(likeStackOverflow)
• KnowledgeBaseArticles
• CodeSamplesandRepositories
![Page 14: Apache NiFi: Ingesting Enterprise Data At Scale](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9ad17760da3da068b94c3/html5/thumbnails/14.jpg)
14 ©HortonworksInc.2011– 2017.AllRightsReserved
CommunityEngagement
Participate now at: community.hortonworks.com©HortonworksInc.2011– 2015.AllRightsReserved
4,000+RegisteredUsers
10,000+Answers
15,000+TechnicalAssets
One Website!