Local Outlier Factor

Lab Goal

Implement Local Outlier factory Batch Mode.

Implement Local Outlier factory Incremental Mode.

Compare two modes.

Integrate code into open source project RealKD:https://bitbucket.org/realKD/

Motivation

http://www.dbs.ifi.lmu.de/Publikationen/Papers/LOF.pdf

Local Outlier Factor

https://en.wikipedia.org/wiki/Local_outlier_factor

Demo

Incremental Outlier Factor

Motivation:

- infinite stream makes memory constraints.
- computational constraint for processing each stream item.

Goal:

- Same performance as iterated static LOF algorithm.
- efficient algorithm: insertion/Deletion should effect only limited number of nearest neighbours

http://www-ai.cs.uni-dortmund.de/LEHRE/FACHPROJEKT/SS12/paper/outlier/pokrajac2007.pdf

Incremental LOF Addition

Berlin;52.520;13.380Hamburg;53.550;10.000Munchen;48.140;11.580Bonn;50.730;7.100Koln;50.950;6.970Frankfurt;50.120;8.680Dortmund;51.510;7.480Stuttgart;48.790;9.190Essen;51.470;7.000Cairo;30.3;31.14Hurghada;27.15;33.50

Incremental LOF Addition

Incremental LOF Addition

1 1.19094756172923642 1.19568308563465563 0.96456311068508184 0.80296014778290055 0.75775401355993616 0.73774956443705167 0.75096085129748678 0.999561011381989 0.694331006095839610 3.7497548217312158 11 3.928514077815152 Now , lets add new Point="Alexandria;31.13;29.58"

Incremental LOF Addition

0 1 2 6 5 8 7 4 3 9 10 1 0 6 5 8 7 4 3 2 9 10 2 7 5 0 1 3 4 6 8 9 10 3 4 8 6 5 7 1 2 0 9 10 4 3 8 6 5 7 1 2 0 9 10 5 7 3 4 6 8 1 2 0 9 10 6 8 4 3 5 7 1 0 2 9 10 7 5 2 3 4 6 8 1 0 9 10 8 6 4 3 5 7 1 0 2 9 10 9 10 2 0 7 5 1 3 4 6 8 10 9 2 0 7 5 1 3 4 6 8

0 1 2 6 5 8 7 4 3 11 9 10

1 0 6 5 8 7 4 3 2 11 9 10

2 7 5 0 1 3 4 6 8 11 9 10

3 4 8 6 5 7 1 2 0 11 9 10

4 3 8 6 5 7 1 2 0 11 9 10

5 7 3 4 6 8 1 2 0 11 9 10

6 8 4 3 5 7 1 0 2 11 9 10

7 5 2 3 4 6 8 1 0 11 9 10

8 6 4 3 5 7 1 0 2 11 9 10

9 11 10 2 0 7 5 1 3 4 6 8

10 9 11 2 0 7 5 1 3 4 6 8 11 9 10 2 0 7 5 1 3 4 6 8

Incremental LOF Addition

Cities 9,10 has change in their K-distance.

According to:
The LRD for cities exists in K-NN of cities (9,10) should updated
LRD List={9,10,2}

According to , all cites that has any of cities {9,10,2} in their new nearest neighbour should update thier LOF value. LOF List={9,10,2,0,7}

Comparison between static and incremental LOF

Running static LOF output:

1.1909475617292364 1.1956830856346556 0.9645631106850818 0.8029601477829005 0.7577540135599361 0.7377495644370516 0.7509608512974867 0.99956101138198 0.6943310060958396 2.3423102537190847 2.342310253719085 2.342310253719085

Running incremental LOF and addition output:1.1909475617292364 1.1956830856346556 0.9645631106850818 0.8029601477829005 0.7577540135599361 0.7377495644370516 0.7509608512974867 0.99956101138198 0.6943310060958396 2.3423102537190847 2.342310253719085 2.342310253719085

Conclusion

Implementation of Batch incremental mode has done.

Batch mode code is integrated into the project repository while pull request has made to integrate it.

Incremental LOF has equivalent detection performance as static LOF.

Incremental LOF requires less computation time than time.

Incremental LOF complexity is O(N log N)

Thank you

Click to edit the title

Click to edit the outline text formatSecond Outline Level Third Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level

Data Mining Lab,Local Outlier FactorAmr Koura / Page Supervisor: Sebastian Bothe