d6.2 - static social orchestration: implementation and evaluation

SmartSociety

Hybrid and Diversity-Aware Collective Adaptive SystemsWhen People Meet Machines to Build a Smarter Society

Grant Agreement No. 600584

Deliverable D6.2 Working Package WP6

Static social orchestration:implementation and evaluation

Dissemination Level(Confidentiality):1

PU

Delivery Date in Annex I: 31/12/2014Actual Delivery Date 18/12/2014Status2 FTotal Number of pages: 47Keywords: compositionality, social orchestration,

abstract architecture

1PU: Public; RE: Restricted to Group; PP: Restricted to Programme; CO: Consortium Confi-dential as specified in the Grant Agreeement

2F: Final; D: Draft; RD: Revised Draft

c© SmartSociety Consortium 2013-2017 Deliverable D6.2

2 of 47 http://www.smart-society-project.eu

Deliverable D6.2 c© SmartSociety Consortium 2013-2017

DisclaimerThis document contains material, which is the copyright of SmartSociety Consortiumparties, and no copying or distributing, in any form or by any means, is allowed withoutthe prior written agreement of the owner of the property rights. The commercial use ofany information contained in this document may require a license from the proprietor ofthat information. Neither the SmartSociety Consortium as a whole, nor a certain party ofthe SmartSocietys Consortium warrant that the information contained in this documentis suitable for use, nor that the use of the information is free from risk, and accepts noliability for loss or damage suffered by any person using this information. This documentreflects only the authors’ view. The European Community is not liable for any use thatmay be made of the information contained herein.

Full project title: SmartSociety: Hybrid and Diversity-Aware CollectiveAdaptive Systems: When People Meet Machines toBuild a Smarter Society

Project Acronym: SmartSociety

Grant Agreement Number: 600854

Number and title of work-package:

WP6 Compositionality and Social Orchestration

Document title: Static social orchestration:implementation and evaluation

Work-package leader: Michael Rovatsos, UEDIN

Deliverable owner: Michael Rovatsos, UEDIN

Quality Assessor: XXX, DFKI

c© SmartSociety Consortium 2013-2017 3 of 47


List of Contributors

Partner Acronym ContributorUEDIN Dimitrios I. DiochnosUEDIN Michael Rovatsos

TUW Ognjen ScekicUOXF Kevin PageUEDIN Pavlos Andreadis



Executive Summary

This document summarises the work performed in WP 6 of the SmartSociety duringthe second year of the project toward achieving a first implementation and evaluationof a static social orchestration architecture based on the description of the deliverableD6.1. We describe the design and implementation of a SmartSociety orchestrator peerthat follows a specific model of orchestration but is generic in terms of the domain inwhich it might be deployed. A preliminary experimental evaluation shows that our designis promising in terms of scalability and robustness. Our report includes extensive detailon actual implementation, including a full description of the orchestration API.



Table of Contents

1 Introduction 8

2 Orchestration Architecture 11

2.1 The SmartSociety Event Loop . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2 Models, Resources, Interactions, and Versioning . . . . . . . . . . . . . . . . 12

2.2.1 Versioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.3 Orchestration Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.4 Platform Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.4.1 Job Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.4.2 IDs, Enqueueing Platform Jobs and Activity Token Update . . . . . 17

2.5 Provenance by Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.6 Priorities, Semaphores and Synchronisation . . . . . . . . . . . . . . . . . . 19

2.6.1 Semaphores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.6.2 Lifecycle of Orchestration Resources . . . . . . . . . . . . . . . . . . 20

2.6.3 Synchronisation Example . . . . . . . . . . . . . . . . . . . . . . . . 21

2.7 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3 Orchestration Workflows 22

3.1 Common Platform Jobs across Workflows . . . . . . . . . . . . . . . . . . . 24

3.2 Asynchronous Read-Only Operations . . . . . . . . . . . . . . . . . . . . . . 24

3.3 Creating Task Requests and Initiating Composition . . . . . . . . . . . . . . 25

3.4 Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.5 Negotiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.6 Deletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.7 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.8 A Default Smart Society Orchestrator . . . . . . . . . . . . . . . . . . . . . 28

4 Evaluation 30

5 Conclusion 32

A Further Operations 33

A.1 Administrative Monitoring Operations . . . . . . . . . . . . . . . . . . . . . 33

A.2 Tunnels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

A.3 Delegated Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34



B Orchestration Patterns 35

B.1 Composition Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

B.2 Deletion Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

B.2.1 Deletion in Full-Negotiation Scenarios . . . . . . . . . . . . . . . . . 35

B.2.2 Deletion in Crowdsourcing Scenarios . . . . . . . . . . . . . . . . . . 36

B.3 Negotiation Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

B.4 Execution Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

C Orchestration API 38

C.1 Application Orchestration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

C.1.1 Task Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

C.1.2 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

C.1.3 Task Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

C.2 Composition Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

C.3 Negotiation Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

C.4 Deletion Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

C.5 Execution Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

C.6 Further Remarks and Functionality . . . . . . . . . . . . . . . . . . . . . . . 46

C.6.1 Administration and Monitoring . . . . . . . . . . . . . . . . . . . . . 47

C.7 The Current Setup in the SmartShare prototype . . . . . . . . . . . . . . . 47



1 Introduction

This report summarises the outcomes of the WP6 work on the implementation and eval-uation of a static social orchestration architecture in year 2 of the project. This workis a continuation of the more conceptual and theoretical investigation into such architec-tures performed in year 1, and documented in deliverable D6.1. The main objective ofboth efforts was to develop methods that allow a lightweight composition of collaborationprocesses in HDA-CAS, which will provide the “scaffolding” which connects componentmethods delivered by WPs 2, 3, 4, and 5 at a conceptual level, and feeds into the sys-tems and software engineering work of WPs 7 and 8 by providing solid conceptual andalgorithmic underpinnings for the software implementations and programming methodsdeveloped in those work packages.

Before proceeding with the exposition of our new research results, it is worth recall-ing the main elements of our abstract model introduced in D6.1 and their motivation.There, we started from a purely abstract model of social computation, where differenthuman and machine computations are performed in a network of interconnected peers inorder to provide an overall systems functionality. Based on this very general model, wethen proceeded to identify a specific architecture that identifies a number of more con-crete computations that can be used to orchestrate such a social computation, i.e. peerdiscovery, task assignment, task execution and feedback. We introduced a mathematicalsignature for this orchestration model, and linked it to the decision-making situation of theautonomous peers involved in the computation, thus relating it to preferences, incentives,and motivations. This provided the blueprint for the orchestration architecture we wouldrealise in subsequent work, and indeed provide in the present report.

The second major contribution of the first phase of WP6 research was the developmentof design principles for social orchestration systems that were intended to maximise scala-bility and robustness. These principles, summarised in what we called the “play by data”architecture provide requirements for orchestration architectures that enable composition-ality in large-scale systems at the level of systems interoperability, communication, andasynchronous, distributed processing.

In this report, we describe the design of a SmartSociety orchestration manager (ororchestrator, for short), which can be viewed both as a (machine) peer that is in chargeof organising social computations using the conceptual model of an orchestration archi-tecture provided in our abstract specification, and as a platform that enables organisingcollaboration among other peers (including task peers that contribute to the executionof the manager peers that contribute functionality to the orchestration process itself) bymanaging the process and data flow between them. This “dual” nature of the orchestratoris important: On the one hand, we view it as an “intelligent software agent” that has tomake decisions regarding its own activities, e.g. which communication requests from othersto serve at what time, which internal and external sub-procedures to trigger under whatcircumstances, and what control structure to apply over them, and how to manage internaland external computational resources. This provides us with ways of producing adaptiveand proactive social orchestration methods in the future (as required for the future tasksof WP6) whose flexibility and range of behaviours exceeds that of traditional workflow



orchestration systems.

On the other hand, by considering it from a “platform-as-a-peer” perspective, it pro-vides an initial design for a “playnode” in the Play-By-Data sense that contains an inte-grated programming interface and communication facility for completing the whole orches-tration cycle and producing complex collaborative tasks. This means that our orchestratorcan be used as a platform that delivers, essentially, the functionality of an entire Smart-Society application through a single Web server which user peers can interact with as“thin” clients (i.e. they can be very minimalistic playnodes that do not offer any servicesthemselves), and which integrates other manager agents (at the moment the peer man-ager, reputation manager and provenance service as external services, and the negotiationmanager and composition manager as internal “sub-orchestration” peers) as simple Webservices. This opens up avenues for lightweight hierarchical and parallel composition ofsocial computations, as we could deploy an arbitrary number of instances of our orches-trators (each with their own different local implementations, based on the general designdescribed in this document) which could, in turn, collaborate on higher-level computationtasks, orchestrated by higher-level orchestrators, etc, within a single architecture. In asense, this results in a “self-similar” design of the overall SmartSociety systems architec-ture, which supports scale-free composition of social computations in HDA-CAS.

It is worth expanding a bit more on this issue, as it is important for understanding thenotion of “architecture” in SmartSociety. In the context of this document (and, in fact,the whole work package), the term (more specifically, orchestration architecture) refersto the data and process model implemented by our orchestration peer, and the sharedcommunication infrastructure assumed for interactions with, within, and between this peerand other peers required for its functionality. In the broader context of the overall project,it refers to the software framework developed by WP8, which enables the integration ofvarious components from individual technical work packages, and over which WP7 willdevelop programming abstractions. In this context, the orchestration architecture will bejust one component provided by WP6 to the overall framework.

Equally fundamental is the notion of “task”, which is closely related to the multi-level compositionality alluded to above. In the context of the orchestration architecture,this is something much more specific than a “general purposeful activity that potentiallyinvolves actions contributed by various peers”; it is the central object that represents acollaborative computation and which is explicitly represented as such in an orchestrator.This means that while there may be other activities in the system like internal operationsof the orchestrator (which we call platform jobs below, e.g. starting a negotiation process)and third-party services used for orchestrations (e.g. a reputation service), these are notexplicitly requested, negotiated, agreed to, or tracked during execution in the same way asa (domain) task, but instead are needed to provide the basic functionality of orchestration.This is not to say at the next level “up” they couldn’t be explicated as tasks themselves(in a reified domain of “orchestration”), but any multi-level hierarchical composition oftasks needs to be “grounded” in a level-0 orchestration backend.

It is this backend functionality that is provided by the orchestrator peer design pre-sented in this document, which is domain-independent but follows our previously definedsingle model of orchestration. Some aspects of this design could be reused, in princi-



ple, to implement any other orchestration model: At the general processing level, ourorchestrator reactively iterates over an asynchronous event processing loop, and serves“jobs” maintained on dynamically adapting process queues. In terms of communication,all interaction is via linked data resources using only basic HTTP operations. Prove-nance tracking, authentication, and access control are enabled for any of these interactionprocesses, regardless of whether they involve user peers or third-party services. What isspecific to our orchestration model are the specific workflows used by the orchestrator,and the decision procedure that determines how processing queues are managed.

In collaboration with TUW and UH, we have adapted this backend functionality sothat the overall design of orchestration and the orchestration API can better integratewith other platform components. This modified design, which is in line with the formalmodel of WP2, is general enough and allows us to cover the two extremes of the socialcomputation spectrum that we can see at the moment: that of full negotiation scenar-ios (as in ridesharing) and that of crowdsourcing scenarios (as in Ask Smart Society!).Further, with the same partners, we have implemented orchestration primitives using theSmartCom communication middleware that is described in D7.1.

It is important to point out that what is not provided in this document is a “workflowmodelling language”, as we focus much more on the implementation and evaluation ofthe orchestration process rather than on the formal modelling of the process. The formalcomputational models for these processes will be provided by WP2, and an early versionthat is fully aligned with our presentation here is given in milestone MS4. This alignmenthas taken place during the development of the SmartShare ridesharing prototype (of whichour orchestrator design is a generalisation, and which we also use for our preliminaryevaluation presented below), and knowledge of the formal model of orchestration presentedthere is necessary to understand the material presented in this document.

It is also worth noting that task execution is not managed in detail yet in the orches-tration stage. This is simply due to the fact that, at the time of writing, more fine-grainedintegration with functionality provided by WP3 and WP5 has not taken place yet (thoughin both cases, arguably some of that functionality concerns more the user interface thanthe orchestration procedure; but see milestone MS14 for further discussion of how this in-formation could be useful for future adaptive orchestration). Thus, some of the workflowsdescribed below serve only as “hooks” for adding this functionality later and have onlybeen realised at a rudimentary level so far. Last but not least, we also investigated alter-native collaboration models appropriate for complex tasks involving large-scale collectivesof human-based peers.

The remainder of this report is structured as follows: In section 2, we introduce theoverall orchestration architecture design in terms of the processing loop it involves, the re-sources it maintains, and the basic data structures and mechanisms it uses for concurrentprocessing and synchronisation. Section 3 describes the workflows used in our orchestra-tion model, how they are managed and synchronised, and includes a “default” design fora simple orchestrator logic that determines how different sub-tasks are dynamically priori-tised based on the current system state by the orchestrating peer. In section 4, we presentpreliminary evaluation results obtained in the ridesharing domain through simulation ex-periments, which mainly assess the scalability and robustness of our platform and have



produced very promising initial results. Three appendices are included at the end of thisreport that provide more implementation-level detail on our current social orchestrationprototype. In appendix A, we cover auxiliary functionality not described in the mainbody of the report. Basic patterns for the various stages of orchestration are described inappendix B. Appendix C, finally, provides a specification of the overall, and a brief de-scription of the components integrated in our implemented SmartShare orchestrator (thathas been tested and evaluated in the ridesharing domain).

2 Orchestration Architecture

We start by giving an overview of the orchestration architecture. Before we delve into thespecifics of orchestration we attempt to give a brief summary of the interactions of peerswith the platform.

The orchestration service is exposed to the rest of the world from specific URIs; seeAppendix C for the API. Every incoming (http) request is translated to a platform job.Each job is loaded to the appropriate orchestration queue. This is the starting point forthe execution of the appropriate workflow so that the job can be served. The orchestratormay have many tasks that are waiting to be served in its queues. Hence, orchestrationrelies on monitoring those queues and deciding from which queue to pick a platform-job.Upon selecting a job from a queue, there is either orchestration-related computation thatneeds to take place, or the job is delegated to an appropriate service for execution. Ineither case, the results of the computation are returned to the orchestrator and dependingon the workflow of the service, a new job may be loaded into one of the orchestrationqueues of the application that is providing the service. Eventually, the appropriate stepof the execution of the workflow is achieved, and the peer/user that posted the originalhttp request to the platform receives the response from the platform.

2.1 The SmartSociety Event Loop

Orchestrating a SmartSociety application happens in an asynchronous event-driven frame-work so that many connections can be handled concurrently. Different orchestrationqueues are associated with different events that can occur in a SmartSociety application.These queues are populated by platform jobs which have all the necessary informationthat is needed for execution as well as for tracing provenance. Typically the queues andthe events (functions) where the computation actually takes place have the same nameunless stated otherwise. Hence, there is an event loop that drives the behaviour of theorchestrator; see Figure 1.

We note that, in principle, at the end of processing of every platform job, the orches-tration manager (or orchestrator, for short) may enqueue more than one platform jobs forfurther processing. Further, some of these jobs might be delegated to other componentsof the platform where the actual job processing is expected to take place. In particular,there are four main sub-components that are used by the orchestrator: the compositionmanager, the negotiation manager, the deletion manager, and the execution manager ; seeFigure 2. We will refer to these managers as sub-orchestration managers.



processfetch job

Orchestrator

Orchestration Queues

Figure 1: The event loop of a SmartSociety application.

Orchestrator

Composition Manager Negotiation Manager Execution ManagerDeletion Manager

Figure 2: The orchestrator and the sub-orchestration managers of a SmartSociety appli-cation

2.2 Models, Resources, Interactions, and Versioning

Below we describe the models and the resources that are used for orchestration. Thedocuments in the different collections serve different purposes. We will now attempt togive a glimpse of the use of these resources and how they are affected by the differentinteractions that are allowed by the system.

Exposed Resources. The exposed resources contain the documents that are neededfor the interactions of peers/clients with a SmartSociety application. These collections aretask requests, tasks, and task records. Figure 3 shows these resources.

The exposed resources allow the core functionality mechanisms for SmartSociety ap-plications; please refer to deliverable D2.2 and milestone MS4 for details on the definitionsof these notions. At a high level, and without loss of generality, task requests can beseen as expected goals for collaborative work among different peers. Composition blendsthese task requests and gives rise to specific tasks where some of the details of the goal



Task Requests Tasks Task Records

Figure 3: Resources exposed by the orchestrator

states may have been altered so that they are actually feasible goal states at the level ofthe collective that is associated with them (interpreted as a team of participants in ourcurrent orchestration model). Thus, negotiation takes place at the level of a task, andthe actual execution of the task is recorded on the relevant task record that has beengenerated by the system once the agreement has been reached. As a side note, changingones preferences happens either at the level of the peer/user-profile (and thus the peermanager is responsible for storing such a change), or at the level of task requests. Further,the resources are designed in such a way that task requests have single owners, while tasksand task records can be accessed only by the members of the collective that appears atthe level of the tasks or task records.

Complementary Resources. The complementary resources contain additional infor-mation regarding the exposed resources. Figure 4 gives an overview.

Task Request

Complements

Task

Complements

Task Record

Complements

Figure 4: The complementary resources that have additional information that might beneeded by the orchestrator or the sub-orchestration managers.

Following our approach outlined in deliverable D6.1, SmartSociety aims to providelightweight applications that follow current standards in Web programming. This impliesthat we want the exposed resources to be able to provide all the relevant information thatis needed to design a client-side logic. However, on the server side, it is often the casethat we need to store additional satellite information for the various documents that areexposed to the clients. For example, the composition manager may want to have handyinformation related to the composition that has been attempted in the past for a specifictask request so that in the future, further composition of that task request with other taskrequests can be sped up. This idea and functionality extends to all the exposed resourcesby allowing a complement to be stored for each one of them.

Auditing Resources. The auditing resources contain information related to the op-erations that were performed by the sub-orchestration managers for the jobs that weredelegated to them. Figure 5 gives an overview of these.



Compositions Negotiations ExecutionsDeletions

Figure 5: The resources that are created by the different sub-orchestration managers. Theinformation contained in them is for auditing and provenance purposes.

These resources, one per sub-orchestration manager, are generated so that we can havea full trace of the outcome of the various computations that take place on the differentsub-orchestration managers, i.e. they can be seen as summaries of these computations. Aproperty

prov

on these documents contains all the relevant information that is needed in order to trackprovenance of the executed processes and their outcomes.

Housekeeping Resources. The housekeeping resources contain information necessaryfor the functionality of the orchestration managers as well as for profiling platform jobsthat are executed on them. Figure 6 gives an overview.

Activity

Tokens

Semaphores Job

Profiles

Queues

Figure 6: Resources that are needed for various housekeeping jobs of the orchestrator aswell as of the sub-orchestration managers.

The largest collection in this set is the collection of job profiles. Documents in the jobprofiles collection contain information regarding the execution of the various jobs; a causalrelationship of the workflow that has been executed up to that particular platform job isgiven together with timings (cpu time, i/o time, queueing time, etc) for all the eventsin the workflow that led to that particular job. Such documents can be very useful foradministrative purposes such as monitoring the behaviour and the responsiveness of thesystem, and can ultimately be used for an adaptive load-balancing mechanisms that canbe implemented on the orchestrator.

Regarding the other resources that fall under this group, only the activity tokensactually persist. The idea is that the various events that are processed by the SmartSocietyapplication are given unique IDs and it is these tokens that store the enumeration of thevarious activities that take place on the system. Thus, these tokens are useful when aSmartSociety application is brought back online, so that the enumeration can continue at



the point were it was taken offline. Such a collection has one document per SmartSocietyapplication, where the various activity tokens are stored in a single document. Finally, forqueues and semaphores all that one really needs is a model of what is supported by eachorchestrator. We will provide further information on these models below. We have onemodel for each case (queues and semaphores) per (sub-)orchestration manager.

2.2.1 Versioning

In some situations it is necessary for the system to store the complete history of a documentthroughout time. For this purpose a property

revision

appears in such documents, indicating which version of the document one is looking at.

By default, all the collections in the exposed resources have equivalent collections withthe versioned history of each document. Thus, for example, in the tasks collection, we canfind the latest version of the tasks in the system, while in the versioned tasks collectionwe can find all the different versions of the tasks of the system.

At present we keep all the versions of the documents that are found in the collectionsfor complementary resources. Such versions allow enhanced auditing mechanisms andperhaps can prove useful for recording further provenance information in the future. Thedocuments in the collections for auditing resources are never versioned, since they arecreated once and never modified by the appropriate sub-orchestration managers. This isalso the case with job profiles since we have one document per activity that has beenexecuted by the system. Finally, we do not version activity tokens.

2.3 Orchestration Queues

The various application queues for orchestration can be divided into the following cate-gories depending on the jobs that need to be served:

1. administration and monitoring, whichare, to a big extent, complementary tothe housekeeping queues (see below),

2. authentication and privacy,

3. delegated services provided by otherpeers, e.g. the reputation service,

4. read-only operations,

5. new task requests (see D2.2),

6. validation,

7. negotiation,

8. deletion,

9. execution,

10. composition, where the task-requestsare used to produce tasks that will benegotiated and eventually executed byhuman or machine peers,

11. changing preferences,

12. consequences of special services, and

13. housekeeping queues.



The above categories of queues naturally imply templates for provenance reasons, i.e.one template for each category. However, the orchestrator actually deals with a finer levelof granularity regarding queues. For example, we have already described two distinctoperations that fall naturally under the housekeeping queues which are (i) update activitytokens and (ii) compute and store platform job profiles. We will comment more on thisas we proceed with our description.

2.4 Platform Jobs

Platform jobs are the elements that are stored in the orchestration queues. Figure 7 givesan example of the translation that occurs in the platform on a URL that is exposed andallows negotiation to be performed on a specific task.

api.put("/applications/:app/tasks/:taskIndex", function (request, response) {

var newJob = {

"application" : app,

"kind" : "authentication",

"id" : activityTokens.authentication,

"purpose" : "negotiateOnATask",

"inputAdapter" : request,

"outputAdapter" : response,

"user" : request.remoteUser.username,

"credentials" : getCredentials(request),

"ip" : getIP(request),

"taskRequestIndex" : null,

"taskRequestVersion" : null,

"taskIndex" : taskIndex,

"taskVersion" : getTaskVersion(request),

"query" : null,

"createdOn" : new Date(),

"enqueued" : new Date(),

"triggeredBy" : null

};

queues [app][newJob.kind].push(newJob);

activityTokens[app][newJob.kind]++;

requestActivityTokensUpdateWhenLoadIsLow(function () {

orchestrator();

});

});

Figure 7: Translating an HTTP request to a platform job

In the simple scenario shown in Figure 7 an HTTP request is translated into a jobfor the platform and is loaded onto the appropriate queue (here, authentication). Theactivity tokens that are associated with each queue guarantee that we can identify thetasks uniquely (see Section 2.2) and thus have the ability to trace the entire history ofa job. Finally, the orchestrator is called since a job is waiting to be served in one of



its queues. In principle, platform jobs give rise to additional platform jobs in the system.Hence, any platform job can be seen as a node in a singly-linked list where the child-parentrelationship is given by the triggeredBy property of each platform job; see Figure 7.

Triggered Jobs. We will be using extensively the term triggered (platform) job toexplain causal relationships in the workflows between different processes. A triggeredjob is not necessarily served immediately. Rather, it is, in the general case, created andenqueued in the appropriate queue. It is always the responsibility of the orchestrator toselect which platform job to serve next, and therefore a call to the orchestrator is madeat the end of every process.

2.4.1 Job Profiling

At the end of execution of every job we can perform job profiling to gather useful infor-mation about a job that has been just served by the SmartSociety platform. For example,we can track error messages sent to a client, or we can identify IPs that need to be black-listed for security purposes. Mostly, however, they will be used to measure processing timefor load-balancing purposes, to identify bottlenecks and for further systems optimisation..Figure 8 gives an example of the information that is stored in the job profiles databasethat is maintained by the system.

2.4.2 IDs, Enqueueing Platform Jobs and Activity Token Update

Every platform job that is executed by a SmartSociety application has a compound keyby which it can be identified uniquely in time. This key can be obtained by looking at theproperties application, kind and ID of each platform job. For every platform job profilethe ID is a string that is the concatenation of the individual values of the compound keyof the respective platform job that is being profiled 3, see, for example, property id inthe JSON document shown in figure 8.

We will now revisit figure 7 to look at the last three commands that are made, andwhich are also shown below.

queues [app][newJob.kind].push(newJob);

activityTokens[app][newJob.kind]++;

requestActivityTokensUpdateWhenLoadIsLow(function () {

orchestrator();

});

In this example, the first line pushes the new platform job in the appropriate queue fororchestration. Since every process that runs on a SmartSociety application has a uniqueID, the second command increments the relevant activity token (integer) for the platformjob that was just enqueued. The last call looks at the queue update activity tokens,and if the queue is empty, creates a platform job indicating that the activity tokens ofthe system have changed and need to be persisted when the load is low 4. Regardless

3the last platform job that appears in the workflow of a platform job profile4In other words, the queue update activity tokens is a queue of size at most 1 at any point in time.



{

"_id": "agreeOnTask-0",

"kind": "agreeOnTask",

"platform_id": 0,

"success": true,

"error": 200,

"errorMessage": "",

"user": "SmartAgent",

"ip": "abc.def.ghi.jkl",

"createdOn": 1405099139505,

"time": {

"other": 3, "profile": 0, "i/o": 41, "cpu": 24, "auth": 396, "queued": 3,

"duration": 467

},

"workflow": [

{ "_id": "53c01c83927b668c1b94e94c", "other": 0, "profile": 0,

"i/o": 0, "cpu": 396, "queued": 1, "duration": 397,

"platform_id": 58, "kind": "authentication"},

{ "_id": "53c01c83927b668c1b94e94b", "other": 0, "profile": 0,


"platform_id": 0, "kind": "validateTaskDocument"},

{ "_id": "53c01c83927b668c1b94e94a", "other": 0, "profile": 0,


"platform_id": 0, "kind": "validateTransition"},

{ "_id": "53c01c83927b668c1b94e949", "other": 0, "profile": 0,


"platform_id": 0, "kind": "agreeOnTask"}]

}

Figure 8: Example of a job profile, for a job in which a user agrees on a task that has beenproposed by the system. Note that this provides a complete breakdown and profiling ofthe workflow that follows until the agreement job is completed.

of whether or not such a platform job is loaded on the update activity tokens queue,the application orchestrator orchestrator is called in the end using a callback. Thus, inthe next step, the orchestrator will decide which queue to dequeue a platform job fromand perform the necessary computation. This is how profiling the platform jobs becomesa crucial operation for the orchestrator and how load balancing enters the picture: theorchestrator acts as an online algorithm that has to perform job prioritisation and adaptto the load of the system.

2.5 Provenance by Design

Platform job descriptions provide all the information that will be used as an input foractually processing the job. Additionally, further information is available in every platformjob so that decisions can be made by the orchestrator as to which jobs to create and prepareto serve. In order to capture provenance at the level of orchestration we need to account



for the following items of information:

• inputs to the various platform jobs,

• ownership and child-parent relationships between jobs,

• unique IDs for the various jobs that are processed,

• time measurements for the various operations while processing platform jobs,

• summaries of the results of the various jobs.

The first two properties are captured in the notion of a platform job. Unique IDs caneasily be created by using activity tokens. Timing measurements are naturally computedfor profiling platform jobs (see section 2.4.1). Finally, the platform jobs together with therest of the resources that are designed and are available to the system capture the resultsand consequences of processing the various platform jobs. All this taken together resultsin a fully provenance-enabled orchestration mechanism.

2.6 Priorities, Semaphores and Synchronisation

As discussed in section 2.3, our orchestration architecture offers a natural partitioningof the different queues into different groups. Each queue is associated with a priorityvalue which indicates how urgent processing the jobs in this particular queue is. Using amachine learning approach like decision tree based learning for dealing with the prioritisednature of the various orchestration queues seems like a natural approach for automatingjob prioritisation, which we might pursue in the future.

In the remainder of this section, we provide general overview of synchronisation mech-anisms that are used in our architecture in the context of various orchestration processes.Later, in section 3 the different workflows that are supported by the orchestrator will bedescribed in a more systematic fashion, and their definitions will lead to an overall spec-ification of a general orchestration algorithm that will allow a SmartSociety orchestratorto deal with the platform jobs that are loaded on the various queues at any point in time.After this, we will combine these components in section 3.8, which presents the “default”concrete decision procedure used by a SmartSociety orchestration manager.

To start off with, we present a somewhat more structured formalisation of the jobselection algorithm. Let Q be the the set of the different orchestration queues such that|Q| = q . Moreover, let pt : Q → N be a priority function where pt(i) indicates the priorityof queue i at time t. Further, at any time t, let `t(i) ∈ N indicate the load of queue iat time t, and let χt(i) = 1 if `t(i) ≥ 1 and 0 otherwise; i.e. χt : Q → {0, 1} an indicatorfunction which is 1 if there is at least one job waiting to be served in the queue at time t.For simplicity we will omit the subscript t in the above functions unless it is not clear fromthe context. Algorithm 1 presents a first approach on selecting a job to be processed fromthe orchestration queues, where pt is a bijection, i.e. all queues have different priorities(the simplest case). In other words, when a call is made to the orchestrator, a platformjob will be processed in the next clock tick if

∑j χ(j) = 1, i.e. if there is at least one

platform job waiting to be served in some queue.



Algorithm 1: A basic job selection algorithm

Input: No input; a call imade to the orchestratorOutput: The queue from which to serve the available job in the next clock tick

1 for i← 0, 1, . . . , (q − 1) do /* iterate over priorities */

2 if χ(p−1(i)

)then return p−1(i); /* p−1 maps priorities to queues */

3 return NIL ; /* No jobs can be served */

2.6.1 Semaphores

Unfortunately, an approach as the one in algorithm 1 is not sufficient for orchestrationpurposes. The associations between task requests, tasks and task records are data-driven(all these resources are represented as linked data) where changes in a document in onecollection may potentially change documents, and this synchronisation has to be managedexplicitly. As a consequence, race conditions can easily arise during the orchestration ofdifferent jobs, and we use semaphores in order to address this issue. A platform job thatis loaded on queue j is said to be a job of type j. Let s(i) ∈ N a semaphore associatedwith queue i ∈ {0, 1, . . . , q − 1} indicating the number of platform jobs of type i that arebeing processed by the system at time t. Further, let c(i) ∈ N∗ for i ∈ {0, 1, . . . , q − 1}the maximum number of platform jobs of queue i that can be processed in parallel by theapplication. One can think of c(i) as the number of cores that are dedicated to serving jobsthat belong to queue i. Instead of querying χ(j) for every queue j as in algorithm 1, wewill use expressions that depend on these parameters and potentially also take parametersreferring to other queues into account. In what follows and unless stated otherwise, wewill assume that c(i) = 1 for i ∈ {0, 1, . . . , q − 1} for simplicity, but our considerationsbelow can be adapted to more computing resources per queue.

The following semaphores are provided in our implementation and can be used fororchestration purposes:

• newTaskRequests

• composition

• negotiation

• deletion

• execution

• updateActivityTokens

We explain their use in the following section.

2.6.2 Lifecycle of Orchestration Resources

In section 2.2 we gave a brief sketch of the interactions that can occur in the system andhow these are associated with various orchestration resources. We will now review some ofthese interactions in order to indicate the necessity and usefulness of the semaphores thatare associated with the various queues. In Section 3 we will examine the workflows thatare supported by the orchestrator and which, taken together, constitute a full descriptionof our generic prototype for orchestrating platform jobs.



Lifecycle of Exposed Resources. The three exposed orchestration resources are shownin Figure 3: Task requests are submitted to the system by peers or users. Assuming thedocuments contain valid information with respect to the application at hand, these willbe stored and trigger a platform job for composition. However, task requests arrive atthe system continuously, and as a result of this, other task requests may be augmentedwith tasks that are the result of future composition platform jobs triggered by future taskrequests. In that sense, task requests are in a composition phase until an agreement ona task has been reached. Moreover, composition takes place in parallel for multiple taskrequests, and it is the orchestrator’s responsibility to synchronise them based on new in-formation that is available either by other composition platform jobs or negotiations thatmay take place in parallel. Once agreement is reached on a task associated with a taskrequest, this task request enters its final state, the agreed phase. After this point executioncan be performed and feedback can be submitted to the agreed task(s).

Tasks are treated in a similar way to task requests. Each task is the outcome of somecomposition job, and can be in five different states: The potential phase indicates thata task satisfies certain “hard” constraints for all the participants involves, but that thequality of the task (as predicted by the orchestrator based on, inter alia, feedback andreputation for the peers involved) does not fulfill the quality expectations of at least oneparticipant. The negotiable phase indicates that a task is ready for negotiation but noneof the involved entities has agreed yet to the task. The under negotiation indicates thatat least one peer has agreed on the task. The agreed phase indicates that the negotiationhas been completed successfully, and the the task requests associated with the task nowtransition from the composition to the agreed. The other terminal state of a task isbecoming invalid. A task can be rendered invalid for many reasons, for example if one ofthe involved participants rejects it during the negotiation process.

Task records contain information on tasks that have been agreed and can be executedin the domain. As the execution phase is not covered by orchestration (yet), we havenot introduced different phases for task records, but most likely, this is either going to beonly one phase, the execution phase, or different (sequential or not) phases that have beengenerated by the orchestrator during the composition of the associated task that is tiedto the particular task record.

Lifecycle of Complementary and Auditing Resources. Figures 4 and 5 the com-plementary and auditing resources generated during orchestration. We do not distinguishphases for these resources, since they either provide complementary information for thevarious exposed resources (which peers cannot interact with directly), or, in the case ofauditing resources, since these are never changed after they are generated.

2.6.3 Synchronisation Example

We are now ready to give an example of synchronising resources and the use of semaphores.Recall that our orchestration platform guarantees that different task requests are stored inthe system in the order in which they arrive and moreover they obtain unique, sequentiallyordered IDs. Now consider the case where an application receives two new task requests



almost simultaneously. These requests need to be persisted. For this reason, a temporaryID is given to the first of them upon arrival and persistence is attempted. However, not allsuch attempts will be successful. For example some task requests may contain malformedinformation, miss data that is important for the underlying model of the task request in thespecific application, etc. The newTaskRequests is used in this process, so that while thisis on, the other task request waits to be served in its queue, which will eventually happenwhen the I/O operation for the first task request finishes. This is important since I/Ooperations are of an asynchronous nature, and without semaphores we would attempt tostore both task requests simoultaneously, but if persisting the first one was not successful,then we would reach a situation where the IDs of the task requests would not be sequentialas expected.

2.7 Scalability

Partitioning platform jobs into different groups suggests possible natural approaches toscaling. For example, jobs that only require read operations can be served by differentmachine peers, given that they all have access to the peer manager database. However,there are jobs that are more involved like, for example, negotiation, where successful agree-ment on a task may have an impact on other, seemingly unrelated, tasks that have beenproduced through composition. In such cases, depending on the platform job workflowsthat we are going to support, we may find some interesting challenges.

3 Orchestration Workflows

This section provides an overview of all individual workflows involved in our orchestra-tion system, which are first described one by one in different sections at a general level,before completing our exposition with the integrated design of a “default” SmartSocietyorchestration peer that integrates them.

In describing workflows we use a graph-based notation as shown in figure 9.

Such graphs use nodes to denote distinct processesing steps in the workflow, e.g. process-A,process-B, profile, and prov in this example. Moreover, there is an orchestration queuewith the same name for every such process that appears in the workflow. The directionof the arrows indicates which platform job triggers which one; for example in Figure 9process-A triggers process-B. In the example, the execution of a process-A involvesthe following steps:

1. Dequeue a platform job from the queue process-A.

2. Record the time of dequeuing and start time of execution of process-A.

3. Perform the operation of process-A, which may involve further asynchronous oper-ations.

4. Upon completion of process-A, record the end time of execution.

5. Create the following platform jobs and load them into the appropriate queues:



profile

↑process-A −→ process-B

↓prov

↓profile

Figure 9: Workflow notation example: once process-A completes execution, process-B istriggered; a process profile is triggered by default on completion of every core workflowprocess like process-A or process-B. A process prov related to tracking provenance istriggered at the end of ever core workflow process in order to track relevant provenanceinformation, which is also profiled by a process profile.

• If there is another step in the main chain of the workflow, prepare the respec-tive job, record the creation timestamp, enqueueing timestamp and enque thisplatform job in the respective queue of the application orchestrator.

• A platform job profile to compute and store the (running-time) profile ofprocess-A that was just served.

• A platform job prov to store the provenance information related to process-A.

6. Upon loading the relevant platform jobs from the previous step enqueue, if necessary,one additional platform job for requesting an update on the activity tokens. Recallfrom Section 2.4.2 that the relevant queue of requesting updates for the activitytokens has size at most one, since the actual update of the tokens in the databaseoccurs only when the load on the system is low.

In the workflows discussed below, we will use the following abbreviations for commonlyused resources:

TR for task requests,

TRC for task request complements,

T for tasks,

TC for task complements,

TD for task records,

TDC for task record complements, and

CR for client (HTTP) requests.



3.1 Common Platform Jobs across Workflows

There is a number of platform jobs that are common among the different orchestrationworkflows. These are:

auth: used for authentication purposes,

prov: used for submitting provenance information to the provenance store,

profile: used for computing the profile of a platform job, and

update activity tokens: used for updating the tokens of various orchestration activities.

At the level of the application orchestrator, auth refers to the authentication thatneeds to take place for the needs of the orchestrator alone. For example, as we willsee in Section 3.3, when a peer posts a task request, the orchestrator first validates theauthentication credentials that were provided together with the task request through thepeer manager. We use auth to refer to such authentication mechanisms that occur forinternal purposes inside the platform. It is worth noting that we do not use queues suchas priv for access control purposes. Rather, the access control mechanism is implicit foroperations that require it; e.g. read a specific task.

Regarding update activity tokens, we will refrain from showing such calls in theworkflows that will follow, as a job is prepared and loaded on the update activity

tokens queue once another job has been loaded in another queue in a strictly sequentialfashion. The reason for this is that loading a job to a queue modifies the activity tokensused by the application, and almost all activities in the system have a unique ID forprovenance reasons. The only exception to this rule are the update activity tokens

jobs themselves.

As concerns priorities for the above queues we assume that

auth < prov < profile < update activity tokens .

This ordering implies that the orchestrator will first attempt to process auth jobs, thenprov jobs, then profile jobs, and finally update activity tokens jobs (when essentiallythe load is going to be very low on the system).

In what follows, similarly to our omission of update activity tokens processes, weomit profile and prov processes for brevity.

3.2 Asynchronous Read-Only Operations

Our architecture allows users and peers to read various resources for purposes of observ-ing aspects of the current state of the process. These read-only operations can occurconcurrently without any further effects on the system. The workflow for such operationsis shown in figure 10. Regarding the priorities on these read-only operations we can beflexible. We choose to treat ETags as more urgent than the actual resources they representand then the resources themselves 5. Beyond these, we also allow users and peers to fetch

5In other words, HEAD operations are considered to be more urgent than GET operations.



CR −→ auth −→ read-only

Figure 10: Workflow for read-only operations

sets of resources in one step, thus requiring only a single authentication in the process (seeappendix C.6). These are typical operations that occur when a client loads for the firsttime. There are many orderings that the above ideas imply, but for simplicity we considerserving sets as more urgent than individual resources (or their ETags) and moreover serv-ing task requests as more urgent than serving tasks. Hence, with ro indicating read-only,we have the following ordering of priorities:

roPersonalTRs < roSetOfTRs < roETagOfTR < roTR < roSetOfTasks < roETagOfTask < roTask .

As mentioned above, the access control mechanism is embedded in the read-only process.

3.3 Creating Task Requests and Initiating Composition

The workflow for creating task requests is shown in figure 11. This has compositiontriggered as a result of a new task request, if this was successfully submitted to thesystem.

CR −→ auth −→ new TR and TRC −→ composition

Figure 11: Workflow for new task requests; a successful response is sent back to the clientat the end of the process new TR and TRC.

The client (HTTP) request CR is authenticated through the process auth. An accesscontrol mechanism is applied inside the process new TR and TRC to check whether this isa permissible operation for the respective client. The process new TR and TRC validatesthe content sent by the client and attempts to persist the new task request TR and itsassociated task request complement TRC. For composition and negotiation reasons, it iscrucial to store satellite information on the database regarding every task request and itis this information that is stored in the complement. Once this process is complete, theresponse is sent back to the client with a link to the task request that the client has justposted and was accepted by the system. The system now triggers composition so thatthe new task request can be matched with other task requests, possibly using additionalinformation from the peer manager and from the reputation service.

In terms of priorities, we have auth < new TR and TRC for this workflow. In termsof synchronisation, as new TR and TRC attempts to lock a computational resource, it willattempt to increase the semaphore newTaskRequests by one so that the actual execution ofthe job can take place. Similarly, at the end of execution, the semaphore newTaskRequestsis decreased by one and a computational resource is released.



3.4 Composition

At the level of the application orchestration, at the moment, the composition can be viewedas a single process as shown in figure 12. However, internally, we have calls defined by the

new TR and TRC −→ composition

Figure 12: Composition workflow, triggered every time a new task request has successfullybeen posted to the system).

orchestration API (see appendix C) where the actual job of composition is delegated toa composition manager. Composition is expected to be a CPU-intensive operation whichmay also inolve calls to the peer manager as well as to the reputation service. In a systemwhere we use only a single processing thread for all orchestration managers, compositionjobs are therefore the least urgent jobs. If on the other hand, at least one dedicatedCPU (or core) is available for composition, delegating the job to the composition managershould have high priority so that the necessary computations can start earlier, in order toensure the system is more responsive.

As regards synchronisation, a composition job will be delegated to the compositionmanager only if the semaphore is below the threshold allowed by the composition manager.This threshold essentially indicates the number of composition jobs that the compositionmanager can process concurrently at any point in time.

The composition manager matches task requests, and it may retrieve information fromthe peer and reputation managers for this purpose (e.g. to only match requests witheach other if the requesting parties “like” each other, or if their user profiles match theconstraints in a request).

This is not covered by our current static orchestration architecture, but initial worktoward building more involved task recommendation algorithms that could be used by anadaptive composition manager is described in milestone MS14. In our currently imple-mentation, composition simply creates all tasks that are possible in principle given thehard constraints listed in requests and returns all of them as potential solutions to therequesting peers.

3.5 Negotiation

The workflow for negotiation is shown in Figure 13.

CR −→ auth −→ negotiation

Figure 13: Negotiation workflow

The negotiation process that takes place in the end of the workflow is shown in figure 13and involves a call to the negotiation manager. Synchronisation is treated in the same way



accept

↗negotiation −→ validate doc −→ validate transition

↘reject

Figure 14: The two possible workflows supporte by the negotiation manager

as for composition (a semaphore called negotiation is used). The negotiation managerexecutes one of the two workflows that are shown in figure 14.

The negotiation manager uses validate doc < validate transition < accept <reject as a priority ordering.

3.6 Deletion

The workflow for deletion is shown in Figure 15.

CR −→ auth −→ deletion

Figure 15: Deletion workflow

The deletion process that takes place at the end of the workflow that is shown in fig-ure 15 involves, per the orchestration API (see Section C), a call to the deletion manager.Synchronisation is managed in the same way as above, using the deletion semaphore.The deletion manager is responsible for handling deletions and synchronising the conse-quences of such deletions. This is further explained in deliverable D7.2. An overview ofits functionality is given in appendix B.2.

3.7 Execution

The workflow for task execution is shown in Figure 16, and it mainly involves a call to theexecution manager (this is very similar to deletion in spirit). The execution semaphore

CR −→ auth −→ execution

Figure 16: Execution workflow

is is used for synchronisation purposes which is called execution. The execution manageris responsible for monitoring execution and potentially synchronising the consequences ofvarious executions. Further information can be found in deliverable D7.2.



3.8 A Default Smart Society Orchestrator

We are now ready to present our “default” SmartSociety orchestrator which assumes thatorchestration runs on a single core of a CPU. Algorithm 2 gives a sketch of the orchestrationprocess for making the decision which platform job to serve on the next clock tick. Apartfrom the processes described above, this involves several further tunneling, delegation, andhousekeeping processes that are described in the annexes. Note that althout the decisionprocedure looks like a list of choices, it actually represents a more complex decision tree,since many queues can be grouped together under one condition, thus requiring at leastone additional branching condition for determining precisely which queue to pick the nextjob from.

Algorithm 2: A sketch of the default orchestrator.

Input: No input; a call is being made to the orchestrator.Output: Determining which platform job to serve on the next clock tick.

1 if (admin or auth job waiting to be served) then process(appropriate job);2 else if (tunnel job waiting to be served) then process(appropriate tunnel job);3 else if (job waiting to be delegated) then process(appropriate delegated job);4 else if (read only job waiting to be served) then process(appropriate read only

job);5 else if (may process a deletion job) then process(appropriate deletion job);6 else if (may process a negotiation job) then process(appropriate negotiation job);7 else if (may process a new TR job) then process(new TR job);8 else if (profile job waiting to be stored) then store(profile job);9 else if (activity tokens update job waiting to be served) then process(update

activity tokens job);10 else if (may compose new job) then process(composition job);

Below we give algorithms that can be used to determine the truth value of the condi-tions in lines 5-7 and 10 this algorithm. More specifically, algorithm 4 presents the methodused for determining the truth value of the condition in line 5. Algorithm 5 describes amethod for determining the truth value of the condition in line 6, algorithm 3 is used todetermine the condition in line 7, and a Algorithm 5 does the same for line 10.

Algorithm 3: Determining whethere a new task request can be processed. Thevariable sTR is a boolean semaphore indicating if a new task request is being served,while `(new TR) denotes the load on the queue for new task requests.

1 if sTR then return false;2 if `(new TR) > 0 then return true;3 return false



Algorithm 4: Determining whether a deletion job can be processed. The variablessD, sN and sC are boolean semaphores used for deletion, negotiation and compositionrespectively, while `(delete) denotes the load on the queue for deletions.

1 if (sD ∨ sN ∨ sC ∨ (`(delete) == 0)) then return false;2 return true

Algorithm 5: Determining whether a negotiation job can be processed. The vari-ables sD, sC and sN are boolean semaphores used for deletion, composition andnegotiation respectively, while `(x) denotes the load the load on queue x.

1 if sD ∨ sC then return false;2 if ¬sN then3 if `(validate doc) > 0 then return true; /* negotiation can start */

4 else return false;

5 else /* already under negotiation */

6 if (`(validate transition) > 0) ∨ (`(reject) > 0) ∨ (`(accept) > 0) then returntrue;

7 else return false;

Algorithm 6: Determining whether a composition job can be processed. The vari-ables sTR, sD, sN and sC are boolean semaphores used for new task requests, deletion,negotiation and composition respectively, while `(composition) denotes the load onthe composition queue.

1 if sD ∨ sN ∨ sC ∨ sTR then return false;2 if (may process new task requests) then return false; /* Algorithm 3 */

3 if (may process deletion) then return false; /* Algorithm 4 */

4 if (may process negotiation) then return false; /* Algorithm 5 */

5 if `(composition) > 0 then return true;6 return false;



4 Evaluation

To establish whether the scalability and robustness improvements we expect from theorchestration model we have developed can be actually observed in real operation, wehave evaluated its prototypical implementation experimentally in the ridesharing domain,which uses the team task orchestration model as described in milestone MS4 of WP2. Ourexperiments below focus on the matchmaking and negotiation part of the protocol, as thisinvolves most dependencies among individual behaviours, and requires involves solvinga complex combinatorial problem for the composition manages that involves calculatingexponential numbers of possible rides presented to every driver and passenger.

For these experiments, we have built a small distributed application that simulatespeers entering the system and posting their tasks. All agents periodically poll the resourcesthey are interested in to check what the currently available rides are, and to determinewhether a ride has been agreed/can no longer be agreed.

In terms of the orchestration platform, our implementation involves a single serverwhich contains deals with all operations on a single CPU. The server runs Node.js6,a non-blocking event-driven JavaScript library, and implements the processing queuesassociated with different stages of orchestration as described above.

Our first experiment examines the overall scalability of the platform. We create arti-ficial “groups” of size k in a population of n peers such that all the task requests inside agroup “match”, and we can artificially control how many rides will be created (i.e. taskshave to be composed in this “matchmaking” process). This experiment involves up to 10groups of 6, 9, and 12 peers, i.e. a total of 60, 90, 120 peers, where the ratio of driversd to passengers p is 1/2 (i.e. p/d ∈ {2/4, 3/6, 4/8} for each group size). Note that therespective number of possible rides generated in each group is (2p − 1) ∗ d as there is adifferent proposal for every subset of passengers, and the rides different drivers may offerto a group overlap. This means that 30/189/1020 rides have to be created for each group,i.e. the system has to deal with up to 10200 rides overall as we keep adding groups. Notealso that, since all ride (=task) requests and agreements to rides occur in very close suc-cession, the load of this system is similar to a real-world system that would experiencethis level of usage every few minutes (in reality, of course, users take much much longer tocheck updates and respond), so it is in fact representative of a very large scale real-worldapplication. Finally, to maximise the amount of messages exchanged and the duration ofnegotiation, drivers accept only the maximally sized ride, and passengers accept all rides.

The top two plots in figure 17 show the average time taken in seconds (across allpeers, and for 20 repetitions for each experiment, with error bars to indicate standarddeviations) for composition and negotiation (all further messages up to the call to theexecution manager), respectively. As can clearly be seen from these plots, even thoughevery user peer has a built-in delay of 2 seconds between any two steps, even when whenthere are 120 peers in the system, the average time it takes a peer to get information aboutall rides acceptable to her/complete the negotiation of a ride is around 50s/80s even inthe largest configurations.

6http://nodejs.org/



Figure 17: Experimental results



In the second experiment, we investigate the cumulative effect of adding delays andmessage failures on the total execution time of an entire negotiation for a ride, in order toassess how robust the system is. For this, we artificially increase the delay between anyupdate a peer receives and its successive operation from 2s to 5s, 10s, and 20s. We usethese artificial delays also to emulate failure, e.g. when network resources are temporarilyunavailable.

The bottom plot in figure 17 shows the results for this experiment, for a group size of 9and 5 groups (45 peers in total), showing measurements for composition, negotiation, andthe total lifespan of an agent (from task request creation to agreement). As can be seen,the overall lifespan of an orchestration increases by a factor of 3 to 4 here when the delayincreases by a factor of 10, which is a good indication that the system degrades gracefullyunder increasing perturbation. Moreover, what is interesting is that the time taken fornegotiation, which involves the highest number of messages to the orchestrator (as allpassengers accept all rides) only increases by a factor between 1.5 and 2. This is becausethe larger delays require less effort for task composition, and the orchestrator has moretime to process negotiation-related messages during these gaps. This nicely illustrateshow separating the processing of different queues leads to effective load balancing for anyorchestrator that has to engage in different interactions concurrently.

5 Conclusion

This document has provided a detailled description of the static social orchestration ar-chitecture developed in SmartSociety. This architecture is purely data-driven, provides aflexible and generic way of organising collcaboration in HDA-CAS, and initial evaluationresults indicate that it results in robust and scalable implementations of SmartSocietysystems.

The design, implementation and evaluation of this architecture completes the firstcycle of research performed in WP6, and provides the foundation for developing adaptiveorchestration architectures in the second cycle.



A Further Operations

Even though basic functionality of Smart Society applications is given by the workflowspresented in the main body of this report, in real-world applications one requires manymore operations to be orchestrated. For completeness, we present these in this appendix.

A.1 Administrative Monitoring Operations

A SmartSociety application should provide monitoring functionalities that are availableonly to the administrators of the application. Examples of such operations that are sup-ported at the moment are the following:

• obtaining an overview of the state of the application, the orchestrator and the indi-vidual resources,

• obtaining specific job profiles,

• retrieving information by using predefined stored procedures; for example retrievinga five number summary of the job profiles for a specific queue, and

• retrieving previous versions of a specific resource regardless of its owner.

The workflows are similar to the asynchronous read-only operations that we describedearlier in section 3.2. The difference is that – apart from versioned task requests, tasks,and task records – the access control mechanisms should only allow administrators of theapplication to perform such operations 7.

Regarding the priorities of the different jobs in this category, we use the following:adminOverview < adminJobProfile < adminStoredProc

adminNumTQs < adminNumTasks < adminNumTQCs

adminversionedTQ < adminversionedTask < adminversionedTQC

. (1)

Finally, we note that the common platform jobs that were described in section 3.1 areessentially administrative jobs that are taken care of automatically by the system.

A.2 Tunnels

A SmartSociety application can provide tunnels to services provided by third parties. Inour current approach we provide such tunnels for authentication and registration of newusers. This means that we allow the clients to communicate directly with the peer managerby simply forwarding requests and responses. The workflow for such operations is shownin figure 18.

7Of course, versioned task requests, versioned tasks, and versioned task records should be accessible notonly by administrators, since these are resources that either belong to a user or a peer, or were generatedby the system and access is allowed to certain users or peers so that the Smart Society application canactually work.



CR −→ tunnel

Figure 18: Tunneling workflow

Regarding the priorities of the two different operations that are allowed through thetunnel (authentication and registration), registration is thought to be more urgent, in asense that responsiveness is more important for new users who join the platform. Thisimplies tunnelregistration < tunnelauthentication for the priorities in tunneling jobs

A.3 Delegated Services

A SmartSharing application may provide serices that are actually delegated to third par-ties. One such instance in our current approach is the reputation service, where theapplication orchestrator plays the role of the middle man between the client and thereputation service. The workflow for such operations is shown in figure 19. In princi-

CR −→ auth −→ delegation

Figure 19: Workflow for delegated jobs

ple one can have a queue for each delegated service regardless of the number of APIcalls that are provided by the-third party service. For illustration purposes, however,we provide an additional refinement on the creation of distinct queues for calls thatare using different HTTP verbs. Here, GET operations are mere read operations andare treated as more urgent than POST operations. Hence, regarding priorities we havedelegatehead < delegateget < delegatedelete < delegateput < delegatepost.

The rationale is similar as in the case of tunnels: At one end of the spectrum wehave HTTP operations such as HEAD and GET, which are expected to be the mostlightweight in terms of required CPU time for processing them, with head operations beingthe most lightweight of the two. At the other end of the spectrum, POST operations areexpected to be the most costly operations since new information is provided with themand further operations might be triggered in the light of this new information. Thus,POST operations are treated as the least urgent. PUT operations can be meaningful fornegotiation protocols for example. However, it is expected that PUT operations are lessCPU-intensive compared to post operations. DELETE operations are considered to bemore urgent than PUT and POST operations since we may actually save CPU time whenPUT/POST operations pending in parallel which otherwise could take into account thedeleted resource. Finally, based on this logic, in the case of multiple delegated services wecluster the priorities according to the above inequalities that are based on the verb thatis being used for the communication, rather than the services themselves.



B Orchestration Patterns

This section introduces specific patterns that are provided by default for orchestration inour current implementation. Note that finalising these patterns within the overall scopeof SmartSociety applications is still work in progress. The material presented here shouldtherefore be read in conjunction with deliverable D7.2.

B.1 Composition Patterns

At the moment we support the composition patterns listed below.

• Create one collective (team) for every subset of matched task requests (i.e. one taskfor each sub-group of participants involved in a task that is a solution for their taskrequests).

B.2 Deletion Patterns

The orchestration service supports the following deletion pattern.

Delete operations are allowed at the level of task requests. Upon receiving such adeletion request, the property

deleted

on the task request complement is incremented by 18.

B.2.1 Deletion in Full-Negotiation Scenarios

Delete operations can be issued by the owners of the task requests or application adminis-trators. The delete operation is accepted, assuming that agreement has not been reachedin a task that is associated with the task request for which deletion has been requested.All the tasks that were associated with the task request that has been deleted are renderedinvalid by changing the type of the links pointing to these tasks to invalid. Note that herewe have an instance of weak consistency, since the documents of the affected tasks do notchange. However, the links on the parent task requests pointing to these tasks are nowcharacterised as invalid.

As a consequence, upon successful deletion, the deletion resource that is created bythe Deletion Manager lists the following:

1. The ID of the task request that was deleted.

2. The error code to be returned to the client.

3. The set of tasks that were rendered invalid (note that some tasks may had alreadybeen invalid in the deleted task request before the deletion took place).

8A task request that has not received a DELETE signal has value 0.



4. The set (not multiset) of the affected task requests. This is the set of all the parenttask requests of all the tasks that were rendered invalid during deletion.

5. A link pointing to the deletion resource that is written down on the task requestcomplement for future retrieval of the outcomes of the delete operation.

In case of unsuccessful deletion, the deletion resource that is created by the DeletionManager lists the following:



Consequences Upon successful acceptance of a delete operation by the deletion man-ager, the orchestrator returns a 404 error code on all subsequent HHTP operations by theresource owner9. However SmartSociety administrators of the application still have accessto the resource for HEAD and GET operations. Similarly, client-side HTTP requests of theform

GET /applications/:app/taskRequests/?peer=:peer

do not include the deleted task requests in the JSON responses. Further, all the tasks ofthe deleted task request are now invalid and thus negotiation (PUT) can no longer takeplace in these tasks. Undelete operations are not allowed by any peer, neither regularpeers nor admin peers.

B.2.2 Deletion in Crowdsourcing Scenarios

The first idea is to follow the paradigm of StackOverflow, where deletions are essentiallyvotes for deletion, and this is why we associate a numerical value to the deleted prop-erty in the complement of the task request. Thus, once the deleted property reaches apre-defined deletion threshold, the task request is locked and only GET operations aresubsequently allowed by regular peers. When the task request becomes locked, then allassociated tasks are characterised as invalid execution stops. Again we have weak consis-tency when the task request enters its locked state.

As a consequence, upon successful deletion, the deletion resource that is created bythe Deletion Manager lists the following:

1. The peer who issued the delete operation.



4. The set of tasks that were rendered invalid (might be empty if the deletion thresholdhas not been reached yet).

9A 403 Forbidden is returned to all other non-admin peers for all HTTP operations anyway.



5. The set (not multiset) of the affected task requests (again it might be empty if thedeletion threshold has not been reached yet). This is the set of all the parent taskrequests of all the tasks that were rendered invalid during deletion.

6. A link pointing to the previous deletion resource (might be null) that is associatedwith the delete operations that are taking place on the specific task request.

7. A link pointing to the deletion resource that is stated in the task request com-plement so that the retrieval of the outcomes of the delete operation can easily beretrieved in the future.

Depending on the desired level of flexibility, we could keep incrementing the counterdeleted forever, or have delete operations be rendered unsuccessful when one attemptsto issue a delete operation on a task request that has reached its deletion threshold andis thus already locked.

Consequences HEAD and GET operations on a locked task request are allowed indef-initely. However, PUT operations are not allowed. The treatment of DELETE operationswill depend on the considerations above.

Finally, undelete operations can be supported if we want to provide such functionality.In this case, the task request complements should be exposed resources to admins, whereGET and PUT operations are allowed only for them.

B.3 Negotiation Patterns

At the moment we support the patterns listed below.

• Any participant of a task may reject the task, thus rendering the task invalid forthe entire collective.

• All the participants in a task agree so that we can finalise the agreement for thecollective.

B.4 Execution Patterns

No specific execution patterns have been defined yet.



C Orchestration API

This final appendix describes the implemented orchestration API.

C.1 Application Orchestration

For convenience we split the API intp different sections.

C.1.1 Task Requests

We start with task requests. The most basic operations are listed in the following table:

verb URI

POST /applications/:app/taskRequests

GET /applications/:app/taskRequests/?user=:user

GET /applications/:app/taskRequests/:taskRequestID

HEAD /applications/:app/taskRequests/:taskRequestID

GET /applications/:app/taskRequests/:taskRequestID/v/:version

DELETE /applications/:app/taskRequests/:taskRequestID

Create Task Request: POST /applications/:app/taskRequests This is the mainURI where new task requests are posted. The JSON object describing the task requestis expected in the body of the request. On success a platform call to the compositionmanager will be made.

Access Control. Any peer or user.

Success. Returns error code 201 together with

• a JSON document of the form

{

data: aURI

}

where aURI is the URI where the client can retrieve (assuming authentication andaccess control policies have no issues) the latest version of the task request that hasbeen posted, and optionally

• an ETag for the JSON object of the response.

Failure. Returns an error code accompanied by an optional error message explainingthe failure; e.g. 403 (forbidden), 500 (internal server error), etc.



Comment. Upon success a call is made to the composition manager that will createthe associated tasks based on the current state of the system.

Get Task Requests of User:GET /applications/:app/taskRequests/?user=:user No parameters are expectedapart from authentication purposes (in the header).

Access Control. The peer user or an admin.

Success. Returns error code 200 together with

• a JSON document of the form

{

data: [[userTaskRequestsURIs], [associatedETags]]

}

that has the list of the task requests referring to the specific user together with theassociated ETags for those links,

• an ETag for the JSON object of the response.

Failure. Returns error code 403 together with the error message ”Forbidden”.

Get a Task Request:GET /applications/:app/taskRequests/:taskRequestID No parameters are ex-pected apart from authentication information (if needed).

Access Control. The owner of the task request or an admin.

Success. Returns error code 200 together with the JSON document of the latestversion of the task request accompanied by the ETag of the document.

Failure. Returns an error code (403, 404) together with an optional error message.

Get the Head of a Task Request:HEAD /applications/:app/taskRequests/:taskRequestID Similar toGET /applications/:app/taskRequests/:taskRequestID except that the body returnedis empty. It just returns the ETag of the latest version of the task request to indicate ifthere has been a change to the document and thus we need to retrieve its latest version.The access control policy is similar as above; the owner of the task request or an admincan perform the operation.



Get a Specific Version of a Task Request:GET /applications/:app/taskRequests/:taskRequestID/v/:version No param-eters are expected apart from authentication purposes (if needed).


Success. Returns error code 200 together with the specific version of the task request.

Failure. Returns an error code together with an optional error message.

Delete a Task Request: DELETE /applications/:app/taskRequests/:taskRequestIDThis is the main URI for deleting task requests. No parameters are expected apart fromauthentication information (in the header). A platform job is prepared and is posted tothe deletion manager.


Success. Returns error code 204.

Failure. Returns an error code (403, 404, or 500).

C.1.2 Tasks

Tasks are generated through composition (more on that below). The most basic operationsrelated to them are listed in the following table:

verb URI

GET /applications/:app/tasks/:taskID

HEAD /applications/:app/tasks/:taskID

GET /applications/:app/tasks/:taskID/v/:version

PUT /applications/:app/tasks/:taskID

Get a Specific Task: GET /applications/:app/tasks/:taskID Similar to GET

/applications/:app/taskRequests/:taskRequestID but referring to tasks. No param-eters are expected apart from authentication information (if needed).

Access Control. The participants of the task or an admin.

Success. Returns error code 200 together with the JSON document of the latestversion of the task accompanied by the ETag of the document.




Get the Head of a Task: HEAD /applications/:app/tasks/:taskID Similar asabove but the body of the response is empty. Essentially this is an easy way for the clientsto figure out if the resource has changed. Same access control policy as above.

Get a Specific Version of a Task:GET /applications/:app/tasks/:taskID/v/:version No parameters are expectedapart from authentication information (if needed).


Success. Returns error code 200 together with the specific version of the task.


Negotiate on a Task: PUT /applications/:app/tasks/:taskID The main callfor negotiation which will trigger an additional platform call to the negotiation manager.Expects the new version of the document of the task taskID. A platform job for negotiationis prepared and is posted to the negotiation manager.


Success. Returns error code 200 together with the new version of the task as isdictated by the negotiation manager.


C.1.3 Task Records

Task records are generated by the orchestrator once execution can start on a specific task.The most basic operations are listed in the following table:

verb URI

GET /applications/:app/taskRecords/:taskRecordID

HEAD /applications/:app/taskRecords/:taskRecordID

GET /applications/:app/taskRecords/:taskRecordID/v/:version

PUT /applications/:app/taskRecords/:taskRecordID



Get a Specific Task Record:GET /applications/:app/taskRecords/:taskRecordID No parameters are expectedapart from authentication information (if needed).


Success. Returns error code 200 together with the json document of the latest versionof the task record accompanied by the ETag of the document.


Get the Head of a Task Record:HEAD /applications/:app/taskRecords/:taskRecordID No parameters are ex-pected apart from authentication information (if needed).

The body of the response is empty. This is another convenience function which allowsan easy way for the clients to figure out if the resource has changed. Same access controlpolicy and error codes as above.

Get a Specific Version of a Task Record:GET /applications/:app/taskRecords/:taskRecordID/v/:version No parametersare expected apart from authentication purposes (if needed).


Success. Returns error code 200 together with the specific version of the task.


Provide Execution Feedback:PUT /applications/:app/taskRecords/:taskRecordID The main call for execu-tion which will trigger an additional platform call to the execution manager. Expects thenew version of the task record document taskRecordID. A platform job for execution isprepared and is posted to the execution manager.


Success. Returns error code 200 together with the new version of the task as dictatedby the execution manager.




C.2 Composition Manager

The composition manager provides the following functionality:

verb URI

POST /applications/:app/compositions

GET /applications/:app/compositions/:compositionID

Perform Composition: POST /applications/:app/compositions Expects the plat-form job with the description, for which the main ingredient is the new task request thathas arrived on the platform.

Access Control. The orchestrator for the application app can make such a call.

Returns. The call always succeeds and generates a resource describing the outcomeof composition. Upon completion it returns an error code 201 and the link to the documentwith the results of composition. Part of the description of the document with the resultsof the composition is the error code and message that is returned through the call POST/applications/:app/taskRequests to the client.

Get Composition Results:GET /applications/:app/compositions/:compositionID No parameters are ex-pected.

Access Control. The orchestrator for the application app or an admin can makesuch a call.

Success. Returns error code 200, the JSON document with the description of theresults of the composition together with the associated ETag for the document.

Failure. Returns an error code (e.g. 404 not found) together with an optional errormessage.

Comment. Normally such a call is expected to happen only once from the applica-tion orchestrator once the latter has received the 201 error code that the composition thatwas requested has been performed.

C.3 Negotiation Manager

The negotiation manager provides the following functionality.



verb URI

POST /applications/:app/negotiations

GET /applications/:app/negotiations/:negotiationID

Perform Negotiation: POST /applications/:app/negotiations Expects the plat-form job with the description, for which the main ingredient is the task on which negoti-ation is being performed.


Returns. The call always succeeds and generates a resource describing the outcomeof negotiation. Upon completion it returns an error code 201 and the link to the documentwith the results of the negotiation. Part of the description of the document with theresults of the negotiation is the error code and message that is returned through the callPUT /applications/:app/tasks/:taskID to the client.

Get Negotiation Results:GET /applications/:app/negotiations/:negotiationID No parameters are expected.


Success. Returns error code 200, the JSON document with the description of theresults of the negotiation together with the associated ETag for the document.


Comment. Normally such a call is expected to happen only once from the applica-tion orchestrator once the latter has received the 201 error code that the negotiation thatwas requested has been performed.

C.4 Deletion Manager

The deletion manager provides the following functionality:

verb URI

POST /applications/:app/deletions

GET /applications/:app/deletions/:deletionID



Perform Deletion: POST /applications/:app/deletions Expects the platform jobwith the description, for which the main ingredient is the task request on which deletionis being performed.


Returns. The call always succeeds and generates a resource describing the outcomeof deletion. Upon completion it returns an error code 201 and the link to the docu-ment with the results of the deletion. Part of the description of the document withthe results of the deletion is the error code that is returned through the call DELETE

/applications/:app/taskRequests/:taskRequestID to the client.

Get Deletion Results:GET /applications/:app/deletions/:deletionID No parameters are expected.


Success. Returns error code 200, the JSON document with the description of theresults of the deletion together with the associated ETag for the document.


Comment. Normally such a call is expected to happen only once from the applica-tion orchestrator once the latter has received the 201 error code that the deletion that wasrequested has been performed.

C.5 Execution Manager

The execution manager provides the following functionality:

verb URI

POST /applications/:app/executions

GET /applications/:app/executions/:executionID

Perform Execution:POST /applications/:app/executions/:executionID Expects the platform job withthe description, for which the main parameter is the task on which a particular step ofthe execution is being performed.




Returns. The call always succeeds and generates a resource describing the outcomeof the execution. Upon completion it returns an error code 201 and the link to thedocument with the results of the execution. Part of the description of the document withthe results of the execution is the error code and message that is returned through the callPUT /applications/:app/taskRecord/:taskRecordID to the client.

Get Execution Results:GET /applications/:app/executions/:executionID No parameters are expected.


Success. Returns error code 200, the json document with the description of theresults of the execution together with the associated ETag for the document.


Comment. Normally such a call is expected to happen only once from the applica-tion orchestrator once the latter has received the 201 error code that the execution thatwas requested has been performed.

C.6 Further Remarks and Functionality

The above are the basic calls for orchestration (ignoring execution). They are designed insuch a way so that resources are generated and linked with each other as needed. We allowa full provenance trace in the above paradigm (for the orchestration part of a SmartSocietyapplication). Further, access control policies allow admins to follow links and inspect thesystem; for example for explanation purposes that can be used for provenance reasons.

In addition, and this is tightly related to privacy and security concerns, the clientsmay want to retrieve sets of task requests or sets of tasks. One approach to accommodatesuch functionality would be calls of the form:

POST /applications/:app/taskRequests/?action=getSet

POST /applications/:app/tasks/?action=getSet

where the client can post a document with the different URIs of tasks or task requests andrequest the content for all of them instead of performing one-by-one calls for retrievingthe individual tasks/task requests. However, such an approach indicates bad design froma REST perspective, so, as we refine and finalise the various details we will allow getoperations on specific resources that can actually guess the functionality that is requestedfrom the server. For example, one functionality that can be provided to the clients is toallow the retrieval of all the task requests that have not been completed yet.



C.6.1 Administration and Monitoring

The orchestration paradigm that we have is asynchronous, non-blocking, and allows fullprofiling of the different steps of the execution of the jobs of the platform. These jobprofiles are stored and administrators of the application/platform can have access to them.Further, the various administrators may require access to resources for inspection purposes.Such monitoring capabilities are offered under URIs of the form shown below.

GET /applications/:app/monitoring/?action=getSet

POST /applications/:app/monitoring

We do provide such functionality and such function calls already 10.

C.7 The Current Setup in the SmartShare prototype

Figure 20 gives a schematic representation of the services and the communication that isallowed at the moment in our prototypical SmartShare implementation.

client

prov

store

peer

manager

reputation

service

orchestration

service

GUI

service

Figure 20: A Schematic representation of the services that are used in SmartShare. Thedashed arrows indicate communication for provenance reasons. The link between the clientand the PROV store is also used for retrieving explanations from the PROV store.

10For example see http://168.144.202.152:3001/monitoring and references therein for versioned re-sources but more importantly the analytics of the platform inhttp://168.144.202.152:3001/monitoring/analytics.
