The algorithm operates as follows:1. Create a set S = { s

1, s

2, …, s

n } representing each office space.

2. Create a kd-tree T using the set S.3. While S is not empty:

a) Pop point s from S and compute the radius r around it containing the nearest 50 neighboring buildings using a pre-built SciPy KDTree and starting maximum distance d = 0.082° 9 km.≅

b) Using T, find all spaces within r and add them to a new cluster, and remove them from S.c) Merge the new cluster into an existing cluster, if there is overlap between them.

4. If the number of clusters is greater than k, recursively perform Step 3-4 with the original set S and 2d as the new maximum distance. Otherwise, merge intersecting clusters and compute a weighted centroid and radius for each cluster.

Creating a Real-Time Recommendation Engine using Modified K-Means Clustering and Remote Sensing Signature Matching Algorithms

AbstractBuilt on Google App Engine, Real-Massive encountered challenges while attempting to scale its rec-ommendation engine to match a 14% week-over-week increase of data. To address this problem of scale, we applied techniques from spectral data processing to trans-form our domain-specific problem. The result: a quantitative solution to a qualitative problem that can match the skill of domain experts while operating in sub-second time.

David Lippa*,† Jason Vertrees**,†

BackgroundSpectral analysis algorithms provide one way to quantify similarity when comparing a data collection against a known signature. This process—material identification3,6,9—is quite literally finding a needle in a pixelated haystack. One such algorithm, Spectral Angle Mapper treats each pixel as an n-dimensional vector, computing the angle between them using the definition of a dot product: A · B = ||A|| ||B|| cos θ. Similarity increases as |θ| approaches 0. 10° is a typical upper threshold. Negative angles are valid in spectral datasets, but not in our case, since values are always positive.

To remap the problem, we:● Treat the list of potential candidates as “pixels” of a spectral data cube.

● Create a library of “signature” vectors.● Cluster using a stripped down version of SciPy's kdtree.py since Google App Engine prohibits execution of native code in 3rd party libraries.2

● Use independent object attributes for vector components, such as cost, size, number of parking spaces, etc.

● Avoid ratios and dependent variables.● Aggregate each cluster's vector components to produce a “signature.”

● Sort the results in ascending order by θ.

This solution results in a quantifiable, accurate, and flexible measurement of similarity.

Phase 1: ClusteringK-means clustering is one of the best-known methods for breaking up n data points into k discrete clusters. While easy to implement and fast in practice, a few worst-case sce-narios may arise in certain unusual data conditions8. To mitigate this, we exploit known attributes of the data: limited overlap between data points since they exist physi-cally in 3-dimensional space; limited data range since the data is clustered by latitude and longitude; related data that can used to improve estimation of the initial cluster sizes.

ResultsSince its inception, the new recommendation service has provided more than 302,925 recommendations in sub-second time. With each call, it sifts through over 80,000 spaces and has handled a workload of 18,327 requests per work day and 6,188 per hour. The result was the product of just 3 weeks of implemen-tation time, from design to production.

In the future, we will add refinements to the clustering algorithm to consider client-specific needs and other related data sets. We can also improve the matching algorithm by applying a cosine rule or Euclidian dis-tance calculation to prevent an extreme case of collinearity–such as the vectors (1, 1, 1) and (1000, 1000, 1000)–showing as a perfect match.

SummaryGoogle App Engine provides a powerful search engine in a scalable infrastructure. It can be customized to address new problems outside of typical keyword searching. To address our problem of pattern matching in commercial real estate, we created a new scalable, domain-specific recommendation engine. We bor-rowed techniques from the field of remote sensing, while also taking advantage of constraints and satisfic-ing over optimizing to overcome our rapid data growth and the restrictions of Google App Engine.

* david.lippa@realmassive.com** jason.vertrees@realmassive.com† RealMassive, Inc. 1717 West 6th St. Austin, TX 78703+ This data cube measures 614 x 512 pixels x 224 bands spanning the entire visible, near-infrared, and short-wave infrared spectrum. Visualizations provided by the open-source Opticks remote sensing toolkit4.

References:1. AVARIS Home page. (2015, June 26). Retrieved from http://aviris.jpl.nasa.gov/data/free_data.html2. Google. (2015, June 11). Google App Engine for Python 1.9.21 Documentation.Retrieved from https://cloud.google.com/appengine/docs/python 3. Landgrebe, David A (2005). Signal Theory Methods in Multispectral Remote Sensing. Hoboken, NJ: John Wiley & Sons.4. Opticks. (2015, June 26). Opticks remote sensing toolkit. Retrieved from https://opticks.org5. RealMassive. (2015, June 10). Retrieved from https://www.realmassive.com

MethodThere are 3 phases needed to overcome constraints imposed by App Engine2:● Cluster user inputs into “signatures” to reduce the length of query strings and sort expressions.● Apply fixed filters to limit search results to within the 10,000 hit sort limit.● Score results by signature match to override the default search-term relevance score.

Doubling the initial radius results in an absolute maximum of 26 recursive calls for an overall asymp-totic complexity of O(2n log

2 n). This never happens in practice due to low building density. The final

result is similar to the representation of clusters in Figure 35. Once the spaces have been clustered, it is trivial to compute the average of each vector component to produce each cluster’s signature.

Figure 3: Clustering 50 spaces from across the US

Figure 2: Graphic representation of hyper-spectral data7

Figure 1: A Commercial Real Estate Survey with Recommendations

Phase 2: FilteringNext, we apply fixed filters informed by domain expertise. For commercial real estate, this includes the building type (e.g. "office", "industrial", etc.), location, and any necessary exclusions. These constraints produce a reasonable subset that can be matched against signatures.

Figure 4: AVARIS data courtesy NASA/JPL-Caltech, showing a signature match1+

6. M. Richmond. Licensed under Creative Commons. Retrieved from http://spiff.rit.edu/classes/phys301/lectures/comp/comp.html7. N Short, Sr. Graphic representation of hyperspectral data. Licensed under Creative Commons.Retrieved from http://rst.gsfc.nasa.gov/ 8. A. Vattani. K-means Requires Exponentially Many Iterations Even in the Plane, Discrete Comput Geom. 45(4): 596–616. 2011.9. H. Zhang, Y. Lan, R. Lacey, W. Hoffmann, Y. Huang. Analysis of vegetation indices derived from aerial multispectral and ground hyperspectral data, International Journal of Agricultural and Biological Engineering. 2(3): 33. 2009.

AcknowledgmentsThe authors would like to thank Fatih Akici, John Leonard, Natalya Shelburne, and Michael Westgate for their suggestions for this poster.

Phase 3: Sort by AngleExecuting the Spectral Angle Mapper algo-rithm on a reduced dataset of 10,000 items equates to performing material identification on a 115 x 87 pixel x 3-band data cube from a multi-spectral sensor, or 3% of the compu-tations required for a small data cube, such as Figure 4. Google App Engine can quickly perform calculations in-place on search re-sults, but it lacks the inverse cosine function2. Our solution uses the cosine ratio as a proxy for the angle: sorting by the cosine ratio in descending order is equivalent to sorting by the angle in ascending order to find the most similar matches.