events.static.linuxfound.org · hdfs audit log 123 addcloseop(file=/a) 124 deleteop(file=/b) 125...
TRANSCRIPT
![Page 1: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/1.jpg)
![Page 2: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/2.jpg)
●
●●
![Page 3: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/3.jpg)
●●●●●●
![Page 4: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/4.jpg)
![Page 5: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/5.jpg)
DFSClient
DFSClient
DFSClient
/user/cmccabe/tpcds_1/user/cmccabe/reports/user/awang/reports
![Page 6: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/6.jpg)
●●●●●
![Page 7: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/7.jpg)
●
●●●
![Page 8: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/8.jpg)
●○○
●○
![Page 9: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/9.jpg)
●
●
●
![Page 10: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/10.jpg)
●●
○●
○
![Page 11: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/11.jpg)
●
Client HDFScreate /adelete /b...
![Page 12: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/12.jpg)
●○○ …
■ …
![Page 13: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/13.jpg)
●
●○
![Page 14: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/14.jpg)
●○○
![Page 15: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/15.jpg)
●○
●●
![Page 16: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/16.jpg)
NameNodeClient
Any events after #123?
Event #124, #125, etc.
Client caches highest event number that it’s seen.
Client polls periodically.
NameNode uses monotonically increasing 64-bit event IDs
![Page 17: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/17.jpg)
●
●
●
![Page 18: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/18.jpg)
HDFS Audit Log
123 AddCloseOp(file=/a)124 DeleteOp(file=/b)125 SetReplicationOp(file=/a,rep=5)126 AddCloseOp(file=/c)127 RenameOp(src=/c, dst=/d)128 ConcatOp(dst=/x, src1=/y, src2=/z)...
DFSINotifyInputStream
123 CreateEvent124 UnlinkEvent125 MetadataUpdateEvent126 CloseEvent127 RenameEvent128 [AppendEvent /x, UnlinkEvent /y, UnlinkEvent /z]...
![Page 19: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/19.jpg)
●
●
○○
●
![Page 20: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/20.jpg)
message RenameEventProto { required string srcPath = 1; required string destPath = 2; required int64 timestamp = 3;}
![Page 21: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/21.jpg)
DFSInotifyEventInputStream stream = dfs.getInotifyEventStream(prevHighestTxId);EventBatch batch = stream.take();
long newTxid = batch.getTxid();switch (batch.getEvents()[0].getEventType()) { case Event.EventType.RENAME:…}stream.close();
![Page 22: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/22.jpg)
●○○
○
●○
![Page 23: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/23.jpg)
●○
●○○○
![Page 24: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/24.jpg)
●
●● …●
![Page 25: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/25.jpg)
●
●
○
![Page 26: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/26.jpg)
●●
●
![Page 27: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/27.jpg)
●●●●
![Page 28: events.static.linuxfound.org · HDFS Audit Log 123 AddCloseOp(file=/a) 124 DeleteOp(file=/b) 125 SetReplicationOp(file=/a, rep=5) 126 AddCloseOp(file=/c) 127 RenameOp(src=/c, dst=/d)](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f64107e708231d443ed1b/html5/thumbnails/28.jpg)
●
●●