hdfs smart storage management...hdfs smart storage management towards higher storage efficiency wei...
TRANSCRIPT
![Page 1: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/1.jpg)
HDFS Smart Storage ManagementTowards Higher Storage Efficiency
Wei Zhou
Apache Big Data Europe 2016
![Page 2: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/2.jpg)
Outline
Motivation
Architecture
Design
Rule
Case Study
Summary
![Page 3: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/3.jpg)
Motivation
Data to be processed and stored boosts Internet of Things Real time stream processing Online Analytical Processing Artificial Intelligence / Deep Learning
Data needs to be processed in time From data been generated to been processed Stored with complex format
![Page 4: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/4.jpg)
Motivation
File size Temperature: hot and cold Workloads: on-line query / off-line analysis
Support for more scenarios
![Page 5: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/5.jpg)
Motivation
Object storage HDFS-7240 Targets at:
Billions of objects Vary for from KB level to tens of MB Reliability, consistency and availability
Object store. No file metadata, K/V based API Supported in Amazon S3, Azure, Aliyun, …
![Page 6: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/6.jpg)
Motivation
Network bandwidth increases 10Gbps network is the mainstream 40Gbps or even 100Gbps is on the way
Hardware
![Page 7: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/7.jpg)
Motivation
More memory
Storage device Cheaper. History data Faster. NVMe and 3D XPoint® Technology
Different types of storage used in HDFS
Hardware
![Page 8: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/8.jpg)
Motivation
But this is not the end of the story!
Facility Target Using
Cache Performance Call API explicitly
Heterogeneous Storage ManagementPerformanceCost saving Call API explicitly
Erasure Coding Space saving Call API explicitly
Mover Maintain Call CLI explicitly
Storage Policy Satisfier Maintain Call API explicitly
DiskBalancer Maintain Call API explicitly
Software
![Page 9: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/9.jpg)
Motivation
But it remains a BIG challenge to identify…
which files need to be cached?
when to un-cache?
Which files to un-cache?
And more like this!!
![Page 10: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/10.jpg)
Motivation
Something that can handle these issues automatically and smartly
by using the right facilities at the right time.
![Page 11: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/11.jpg)
Motivation
Key to these questions sense the data temperature timely predicate the temperature change deal with the change evaluate a storage device’s efficiency
![Page 12: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/12.jpg)
Motivation
Aware of current status
States of resources
States of data
To solve these question, we have to:
Respect to users
Definition and threshold
Preference
Learn from history
access pattern
![Page 13: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/13.jpg)
Motivation
UserConfig
Current
Status
HistoryInfo
Algorithm Action
![Page 14: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/14.jpg)
Architecture
…
Name Node 1
SSM
Data Node 1
…
Data Node N
Admin
Name Node MExecute action
Info
Client
Managerules
Query info
Info
![Page 15: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/15.jpg)
Principle
Optional service for HDFS Run facilities manually may not be allowed
Should not: Break the function of cluster Bring in security issue to the cluster
Trying to: Minimize the overhead to the cluster be simple for porting
Before we dive into the detailed design:
![Page 16: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/16.jpg)
Design
Interface with user/admin
![Page 17: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/17.jpg)
Design
SSM
Data query
Data query
Dataquery
Savedata
Checkpoint
Events
Events Events
Actions Actions
ActionExecutor
StatesManager
CacheManager StorageManager
RuleManager H D
F S
Local Sto
rage
StatesManager
Historical information. For example, file access history, cache hit statistics, disk throughputs of DataNodes.
Current status information. E.g. file storage policy, a file is in cache or not. This kind of information is not required to be stored as it can be queried from NameNodes when needed.
Forward and generate events to RuleManger
![Page 18: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/18.jpg)
Design
SSM
Data query
Data query
Dataquery
Savedata
Checkpoint
Events
Events Events
Actions Actions
ActionExecutor
StatesManager
CacheManager StorageManager
RuleManager H D
F S
Local Sto
rage
RuleManager
Parse rules and execute rules Explore rule for files without
specifying a rule. Templates
![Page 19: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/19.jpg)
How to use of SSM?
SSMRule
…Name Node 1
Data Node 1
…
Data Node N
Name Node M
HDFS Cluster
Achieve better performance without modifying upper App logic
Design
![Page 20: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/20.jpg)
Rule
# comment starts with a ‘#’Objects_to_manipulate:[ Trigger | ] Conditions | Commands
Syntax
It links history info, current status, user configuration and action together. It’s a guide line for SSM to function.
Objects:filedirectorystoragecachememorynodecluster
Trigger: when the ‘Conditions’ are evaluated and checked. It’s optional.
It combines the history, current status and user preference together.Predefined variables be used to setup the expression
What SSM should do when the ‘Conditions’ fulfilled
![Page 21: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/21.jpg)
Rule
file.path matches “/fooA/abc*”:accessCount(10min) >= 10 | cache
Examples
datanode:every 1:00 | datanode.storageUnbalanceRatio(‘SSD’) > 30 | diskbalance
file.path matches “/fooB/*”:age >= 30d | archive
![Page 22: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/22.jpg)
Rule
StatesManager RuleManager CacheManager StorageManager ActionExecutor
Analysis RuleNotify the kind of data needed
Notify
Trigger?
Wait for event
Yes
Periodical
Check cond.Check cond.
No
Gen. event Gen. event
Yes Yes
NoNoActions
Event ?
Execution flow
![Page 23: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/23.jpg)
Case Study
DataNode
HDD
SSD
MEM
B
ReadDataNode
HDD
SSD
MEMB
Read
B
file.path matchs “/foo/*”:accessCount(10min) >= 3 | mover ONE_SSD
DataNode
HDD
SSD
MEMB
Read
B
file.path matchs “/foo/*”:accessCount(10min) >= 3 | cache
Optimize when getting HOT
![Page 24: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/24.jpg)
Case Study
Without SSMIt’s hard to implement!
Archive COLD data
With SSM
Archive when the cluster is in low load
file.path matchs “/foo/*”:age > 30d | archive
COLD data: files under directory /foo and age larger than 30 days
![Page 25: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/25.jpg)
Case Study
Without SSMIt’s hard to implement!
Archive COLD data
With SSM
Archive when the cluster is in low load
file.path matchs “/foo/*”:accessCount(30d) < 3 | archive
COLD data: files under directory /foo and not been read for more than 3 times in last 30 days
![Page 26: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/26.jpg)
Case Study
Client NameNode
SSM
3. create file with storage policy “ALL_SSD”
Optimization on write with fast storage
![Page 27: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/27.jpg)
Status
The discussion is continuing on
Prototype undergoingImplementation for the 3 use casesArchive cold dataMove hot data to fast storageCache hot data
![Page 28: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/28.jpg)
Status
Enhance HDFS cache for partial caching
Block-level statistics and optimization
Extend EC for data archive usage
![Page 29: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/29.jpg)
Summary
Rule-based engine State-aware managementAutomationProvide an unified interface to user Flexible Tune HDFS to fit application behaviors
We introduce in an mechanism to optimize the efficiency of HDFS cluster:
JIRA: HDFS-7343Any suggestions or participations will be appreciated!
![Page 30: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/30.jpg)
![Page 31: HDFS Smart Storage Management...HDFS Smart Storage Management Towards Higher Storage Efficiency Wei Zhou Apache Big Data Europe 2016 Outline Motivation Architecture Design Rule Case](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec9a3d866b0bc5d441a5c75/html5/thumbnails/31.jpg)
Legal Disclaimer
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.
Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.
*Other names and brands may be claimed as the property of others.
Copyright © 2016 Intel Corporation.
31