in stream big data processing
DESCRIPTION
Cloud platforms integrate three pillars: messaging, coordination of workers and data. This research investigates whether functional programming languages (FPLs) have any special merit when it comes to the implementation of cloud computing platforms. We have implemented an optimistic message queue (CMQ) and an in memory tuple space (cwmwl) for the coordination of workers that can be used as artefact to proof or disproof the special merit of FPLs in computing clouds. We now address the problem of in-stream processing of Big Data (for example social network streams, logistic data or data mashups) to investigate the integration with Big Data.TRANSCRIPT
In stream Big Data processing.Joerg Fritsch, [email protected] of Computer Science and Informatics,Cardiff University UK
IntroductionCloud platforms integrate three pillars: messaging, coordination of workers and data. This research investigates whether functional programming languages (FPLs) have any special merit when it comes to the implementation of cloud computing platforms. We have implemented an optimistic message queue (CMQ) and an in memory tuple space (cwmwl) for the coordination of workers that can be used as artefact to proof or disproof the special merit of FPLs in computing clouds. We now address the problem of in-stream processing of Big Data (for example social network streams, logistic data or data mashups) to investigate the integration with Big Data.
Hypothesis Functional Programming Languages (FPLs), with their inherent parallelism and easy-access, lightweight units of scale, working together with a LINDA coordination model, provide an ideal basis for a cloud computing platform (PaaS) consisting of messaging, coordination and the in-memory integration with data (flows).
Optimistic messaging, coordination and data.Figure 1 applies the principles learned from FPLs (e.g. shared nothing, modularity, no state) to create a framework for in transit processing of big data.
Implications of our workIaaS, PaaS and SaaS are today’s means to create computing clouds, --the illusion of unlimited elastic computing resources. Map Reduce (MapR) are today’s answer to the processing of Big Data. However, both worlds ( _aaS and MapR) are sold as separate offerings because they do not integrate with each other and none of the solutions are truly elastic. Our work holds special significance in the development of true elasticity by minimizing the units of scale and in the integration of the provisioned units of scale with real time (big) data processing.
PapersFritsch J. Walker C. (2012), “CMQ - A lightweight, asynchronous high-performance messaging queue for the cloud”, Journal of Cloud Computing: Advances, Systems and Applications, 1(1)Fritsch J. Walker C. (2013), “Cwmwl, a LINDA-based PaaS fabric for the cloud”, Journal of Communications, SI on Cloud and Big Data (to be published)
Figure 1Logical Diagram showing combination of optimistic messaging and the in memory tuple space for in stream data processing. The functional programming language Lua is used for data processing such as data masking or “scrubbing”.