adobe aem - from eventing to job processing
DESCRIPTION
Evolve13 presentation from OSGi Event Admin, Sling Discovery, Sling Job Handling to Granite Offloading and AEM Dam IngestionTRANSCRIPT
![Page 1: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/1.jpg)
FROM EVENT ING TO JOB PROCESS ING
Carsten Ziegeler | Adobe Research Switzerland
1
![Page 2: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/2.jpg)
2
• RnD Team at Adobe Research Switzerland • Co-founder Adobe Granite
• OSGi Core Platform and Enterprise Expert Groups • Member of the ASF
• Current PMC Chair of Apache Sling • Apache Sling, Felix, ACE
• Conference Speaker • Technical Reviewer • Article/Book Author
[email protected] @cziegeler
ABOUT
![Page 3: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/3.jpg)
3
OVERVIEW Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
Fro
m E
ven
ting
to
Jo
b P
roc
ess
ing
![Page 4: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/4.jpg)
4
OVERVIEW Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
Fro
m E
ven
ting
to
Jo
b P
roc
ess
ing
![Page 5: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/5.jpg)
5
OSGI EVENT ADMIN Publish / Subscribe Model
OSGi Event Admin Component
A
publish deliver
Component X
Component Y
![Page 6: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/6.jpg)
6
• OSGi event is a data object with • Topic (hierarchical namespace) • Properties (key-value-pairs)
• Page Event • Topic: com/day/cq/wcm/core/page • Properties: path, change type (add/remove/edit) etc.
OSGI EVENT ADMIN Publish / Subscribe Model
![Page 7: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/7.jpg)
7
• Publisher creates event object • Sends event through EventAdmin service • Either sync or async delivery
• Subscriber is an OSGi service (EventHandler) • Service registration properties • Interested topic(s)
• com/day/cq/wcm/core/* • Additional filters (optional)
• (type="add“)
OSGI EVENT ADMIN Publish / Subscribe Model
![Page 8: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/8.jpg)
8
• Immediate delivery to available subscribers • No guarantee of delivery • No distributed delivery
OSGI EVENT ADMIN Publish / Subscribe Model
![Page 9: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/9.jpg)
9
• Immediate delivery to available subscribers • No guarantee of delivery • No distributed delivery
OSGI EVENT ADMIN Publish / Subscribe Model
Discovery Sling Job Distribution
![Page 10: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/10.jpg)
10
OVERVIEW Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
![Page 11: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/11.jpg)
11
OVERVIEW Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution New in 5.6.1
![Page 12: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/12.jpg)
12
INSTALLAT ION SCENARIOS
Clustered CRX
Single Instance
CRX
Instance 1
Instance 2
Instance 3
![Page 13: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/13.jpg)
13
Instance: Unique Id (Sling ID)
TOPOLOGIES I Apache Sling Discovery
Clustered CRX CRX
ID : A ID : X ID : 42 ID : 1
Single Instance
Instance 1
Instance 2
Instance 3
![Page 14: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/14.jpg)
14
• Instance: Unique Id (Sling ID) • Cluster: Unique Id and leader
TOPOLOGIES I Apache Sling Discovery
Cluster 99
Cluster 35
Clustered CRX CRX
ID : A ID : X ID : 42 ID : 1
Single Instance
Instance 1
Instance 2
Instance 3
Leader Leader
![Page 15: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/15.jpg)
15
• Topology: Set of clusters
TOPOLOGIES I Apache Sling Discovery
Cluster 99
Cluster 35
Clustered CRX CRX
ID : A ID : X ID : 42 ID : 1
Single Instance
Instance 1
Instance 2
Instance 3
Leader Leader
Topology Topology
• Instance: Unique Id (Sling ID) • Cluster: Unique Id and leader
![Page 16: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/16.jpg)
16
TOPOLOGIES I Apache Sling Discovery
• Instance: Unique Id (Sling ID) • Cluster: Unique Id and leader
• Topology: Set of clusters
Cluster 99
Clustered CRX CRX
ID : A ID : X ID : 42 ID : 1
Single Instance
Instance 1
Instance 2
Instance 3
Leader
Topology
Cluster 35
Leader
![Page 17: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/17.jpg)
Cluster 99
17
• Instance • Sling ID • Optional: Name and description • Belongs to a cluster • Might be the cluster leader • Additional properties which are distributed
• Extensible through own services (PropertyProvider) • E.g. data center, region or enabled job topics
• Cluster • Elects (stable) leader • Stable instance ordering
• TopologyEventListener: receives events on topology changes
TOPOLOGIES I I Apache Sling Discovery
ID : 42
Instance 3
Topology
Leader
![Page 18: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/18.jpg)
18
OVERVIEW Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
![Page 19: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/19.jpg)
19
• Job : Guaranteed processing, exactly once • Exactly one job consumer
• Started by client code, e.g. for replication, workflow... • Job topic • Payload is a serializable map
JOB HANDLING I Apache Sling Job Handling
![Page 20: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/20.jpg)
20
• Sling Job Manager handles and distributes jobs • Delivers job to a job consumer… • …and waits for response • Retry and failover
• Notification listeners (fail, retry, success)
JOB HANDLING I Apache Sling Job Handling
![Page 21: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/21.jpg)
21
S TART ING / PROCESS ING A JOB I Apache Sling Job Handling
public interface JobConsumer { String PROPERTY_TOPICS = "job.topics"; enum JobResult { OK, FAILED, CANCEL, ASYNC } JobResult process(Job job);}
public interface JobManager { Job addJob(String topic, String optionalName, Map<String, Object> properties); …}
Starting a job
Processing a job
Note: Starting/processing of jobs through Event Admin is deprecated but still supported
New in 5.6.1
![Page 22: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/22.jpg)
22
@Component@Service(value={JobConsumer.class})@Property(name=JobConsumer.PROPERTY_TOPICS, value="org/apache/sling/jobs/backup")public class BackupJobConsumer implements JobConsumer { @Override public JobResult process(final Job job) { // do backup return JobResult.OK; }}
S TART ING / PROCESS ING A JOB I I Apache Sling Job Handling
New in 5.6.1
![Page 23: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/23.jpg)
23
• New jobs are immediately persisted (resource tree / repository) • Jobs are “pushed” to the processing instance • Processing instances use different queues
• Associated with job topic(s) • Main queue • 0..n custom queues
• For example: replication agent queue or workflow queue
JOB HANDLING I I Apache Sling Job Handling
![Page 24: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/24.jpg)
24
• Queue is configurable • Queue is started on demand in own thread
• And stopped if unused for some time • Types
• Ordered queue (eg replication) • Parallel queues: Plain and Topic Round Robin (eg workflow)
• Limit for parallel threads per queue • Number of retries (-1 = endless) • Retry delay • Thread priority
JOB QUEUE Apache Sling Job Handling
![Page 25: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/25.jpg)
25
• Job Manager Configuration = Main Queue Configuration • Maximum parallel jobs (15) • Retries (10) • Retry Delay
• Eventing Thread Pool Configuration • Used by all queues • Pool size (35) = Maximum parallel jobs for a single instance
ADDIT IONAL CONFIGURAT IONS Apache Sling Job Handling
![Page 26: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/26.jpg)
26
MONITORING – WEB CONSOLE Apache Sling Job Handling
![Page 27: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/27.jpg)
27
MONITORING – WEB CONSOLE Apache Sling Job Handling
![Page 28: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/28.jpg)
28
• Each instance determines enabled job topics • Derived from Job Consumers (new API required) • Can be whitelisted/blacklisted (in Job Consumer Manager) • Announced through Topology
• Job Distribution depends on enabled job topics and queue type • Potential set of instances derived from topology (enabled job topics) • Ordered : processing on leader only, one job after the other • Parallel: Round robin distribution on all potential instances
• Local cluster instances have preference
• Failover • Instance crash: leader redistributes jobs to available instances
• Leader change taken into account
• On enabled job topics changes: potential redistribution
JOB DISTR IBUT ION Apache Sling Job Handling
![Page 29: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/29.jpg)
29
• Scalability in AEM: • DAM Ingestion • Non-Clustered installation requirement
• The term Offloading: • In AEM used for all things job distribution and topology in clustered and non-
clustered installations, e.g. ‘Offloading Browser’ • More technically it’s ‘only’ a little add-on in Granite to Sling Job Distribution
for handling non-clustered installations
WHY OFFLOADING
![Page 30: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/30.jpg)
30
OVERVIEW Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
![Page 31: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/31.jpg)
31
OVERVIEW Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
![Page 32: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/32.jpg)
32
SL ING JOB DISTR IBUT ION
Topology
Instance Sling ID: 1
Job Manager
Instance Sling ID: 2
Job Manager
Instance Sling ID: 3
Job Manager
Instance Sling ID: 4
Job Manager
Job Consumer Topic: A
Job Consumer Topic: B
Job Consumer Topic: C
![Page 33: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/33.jpg)
33
SL ING JOB DISTR IBUT ION
Instance Sling ID: 1
Job Manager
Instance Sling ID: 2
Job Manager
Instance Sling ID: 3
Job Manager
Instance Sling ID: 4
Job Manager A
Job Consumer Topic: A
Job Consumer Topic: B
A:2
Job Consumer Topic: C Job
Topology
![Page 34: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/34.jpg)
34
SL ING JOB DISTR IBUT ION
Instance Sling ID: 1
Job Manager
Instance Sling ID: 2
Job Manager
Instance Sling ID: 3
Job Manager
Instance Sling ID: 4
Job Manager B
Job Consumer Topic: A
Job Consumer Topic: B
B:3
Job Consumer Topic: C Job
Topology
![Page 35: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/35.jpg)
35
SL ING JOB DISTR IBUT ION
Instance Sling ID: 1
Job Manager
Instance Sling ID: 2
Job Manager
Instance Sling ID: 3
Job Manager
Instance Sling ID: 4
Job Manager C
Job Consumer Topic: A
Job Consumer Topic: B
C:4
Job Consumer Topic: C Job
?
Topology
![Page 36: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/36.jpg)
36
SL ING JOB DISTR IBUT ION
Instance Sling ID: 1
Job Manager
Instance Sling ID: 2
Job Manager
Instance Sling ID: 3
Job Manager
Instance Sling ID: 4
Job Manager C
Job Consumer Topic: A
Job Consumer Topic: B
C:4
Job Consumer Topic: C Job
Offloading Framework
Topology
![Page 37: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/37.jpg)
37
• Detects offloading jobs • Transport of job and job
payload between origin and target instance
• Uses replication for the transport
• No distribution of jobs • No execution of jobs
OFFLOADING FRAMEWORK Overview
Instance Sling ID: 1
Job Manager
Offloading Job Manager
C:4
Instance Sling ID: 4
Job Manager
Job Consumer Topic: C
Offloading Job Manager
C:4
C
Job
Replication
1
2
3 5
4
6 7
Topology
![Page 38: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/38.jpg)
38
• Replication agents are created automatically • Uses naming convention • Needs manual adjustments (replication user)
OFFLOADING FRAMEWORK Replication
Worker Sling ID: 4
Author Sling ID: 1
offloading_4 (outgoing agent)
offloading_reverse_4 (reverse agent)
offloading_outbox (outbox agent)
![Page 39: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/39.jpg)
39
• Additional properties required • Offloading job input
• Property: OffloadingJobProperties. INPUT_PAYLOAD (offloading.input.payload) • Offloading job output
• Property: OffloadingJobProperties. OUTPUT_PAYLOAD (offloading.output.payload)
• Takes comma separated list of paths • Used to build the replication package • Job sender needs to set these properties for offloading to work!
OFFLOADING FRAMEWORK Job Payload
![Page 40: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/40.jpg)
40
• Configures Job Consumers • Configures the topic white/black listing properties of each instance • What jobs to execute on what instance
• Configures the distribution • Configuration applies for both, clustered and non-clustered installations
OFFLOADING BROWSER UI
![Page 41: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/41.jpg)
41
OFFLOADING BROWSER UI
![Page 42: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/42.jpg)
42
OFFLOADING BROWSER UI
![Page 43: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/43.jpg)
43
OFFLOADING BROWSER UI
![Page 44: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/44.jpg)
44
OVERVIEW Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
![Page 45: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/45.jpg)
45
• New JobConsumer • Class: WorkflowOffloadingJobConsumer • Topic: com/adobe/granite/workflow/offloading • Can launch new workflows • Expects the workflow model on the job payload • Expects the workflow payload on the job payload
• For use with clustered and non-clustered installations
WORKFLOW DISTR IBUT ION
![Page 46: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/46.jpg)
46
@Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } }
WORKFLOW DISTR IBUT ION Job Consumer (Simplified)
![Page 47: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/47.jpg)
47
@Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } }
WORKFLOW DISTR IBUT ION Job Consumer (Simplified)
• Define the job topic
![Page 48: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/48.jpg)
48
@Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } }
WORKFLOW DISTR IBUT ION Job Consumer (Simplified)
• Create service component • Must register with topic • Implement new JobConsumer
interface
![Page 49: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/49.jpg)
49
@Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } }
WORKFLOW DISTR IBUT ION Job Consumer (Simplified)
• Access job properties (payload) • Read workflow model and
payload from job properties
![Page 50: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/50.jpg)
50
@Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } }
WORKFLOW DISTR IBUT ION Job Consumer (Simplified)
• Workflow specific • Use workflow API to start
workflow for the given model and payload
![Page 51: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/51.jpg)
51
@Component @Service @Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC) public class WorkflowOffloadingJobConsumer implements JobConsumer { public static final String TOPIC = "com/adobe/granite/workflow/offloading"; public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model"; public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”; public JobResult process(Job job) { // read workflow model and payload from job payload String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , ""); String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , ""); // get/create WorkflowSession, WorkflowModel and WorkflowData objects WorkflowSession wfSession = ..; WorkflowModel wfModel = ..; WorkflowData wfData = ..; // start the workflow wfSession.startWorkflow(wfModel, wfData, metaData); // all good return JobResutl.OK; } }
WORKFLOW DISTR IBUT ION Job Consumer (Simplified)
• Use JobResult enumeration to report back the job status
![Page 52: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/52.jpg)
52
OVERVIEW Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
![Page 53: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/53.jpg)
53
• Default ingestion workflow: “DAM Update Asset” • Load is put on the instance where the workflow is started, usually the author
• New ingestion workflow: “DAM Update Asset Offloading” • Needs to be manually enabled by changing the workflow launcher • New workflow model with a single step: AssetOffloadingProcess • Uses WorkflowExternalProcess API • Creates a new job on topic: com/adobe/granite/workflow/offloading • Allows distributing the default ingestion workflow • Load is put on the instance where the job is distributed to
• Can be used to distribute in clustered and non-clustered installations
DAM INGEST ION
![Page 54: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/54.jpg)
54
@Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); }
DAM INGEST ION Create Job (from workflow step)
![Page 55: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/55.jpg)
55
@Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); }
DAM INGEST ION Create Job (from workflow step)
• Create service component • Implement
WorkflowExternalProcess interface
• Reference JobManager service
![Page 56: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/56.jpg)
56
@Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); }
DAM INGEST ION Create Job (from workflow step)
• DAM and Workflow specific • Resolve to Asset • Read model from meta data • Read workflow payload from
Asset path
![Page 57: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/57.jpg)
57
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); }
DAM INGEST ION Create Job (from workflow step)
• ValueMap for job properties • Put model and payload on job
properties • Used by the JobConsumer
![Page 58: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/58.jpg)
58
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); }
DAM INGEST ION Create Job (from workflow step)
• Build offloading payload properties • Comma separated list of paths • Put them on the job payload as
well • Only used for non-clustered
distribution
![Page 59: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/59.jpg)
59
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ Asset asset = ..; String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”; String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”; ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>()); jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel); jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload); String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model, /content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ; jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput); jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput); Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties); return offloadingJob.getId(); }
DAM INGEST ION Create Job (from workflow step)
• Create job using JobManager service
• Use topic from job consumer • Put job payload properties • Return the jobId as the workflow
process id (workflow specific)
![Page 60: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/60.jpg)
60
@Component @Service public class AssetOffloadingProcess implements WorkflowExternalProcess { @Reference private JobManager jobManager; private static final String TOPIC = "com/adobe/granite/workflow/offloading"; public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){ …. } public boolean hasFinished(Serializable externalProcessId, ..){ // returns null, if job is finished Job offloadingJob = jobManager.getJobById((String) externalProcessId); return offloadingJob == null; } }
DAM INGEST ION Create Job (from workflow step)
• Workflow API specific callback • Process id = jobId, from
execute() • Query job by jobId • Workflow step finished when job
is finished
![Page 61: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/61.jpg)
61
• Choose a job topic • Create JobConsumer component and register with topic chosen • To create a new job use new JobManager.addJob() API with the topic chosen
and the job payload • Add offloading payload to job payload • Bundle and deploy JobConsumer on topology instances • Enable/Disable the new topic on the instances, using Offloading Browser
DEVELOPMENT - RECIPE
![Page 62: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/62.jpg)
62
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
![Page 63: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/63.jpg)
63
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
org.apache.sling.discovery.api • Discovers topology • Cross-Cluster detection • Foundation for job distribution
![Page 64: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/64.jpg)
64
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
org.apache.sling.event • Uses Sling Discovery • New JobConsumer API and job topics • New JobManager API for creating new distributed jobs • Distributes jobs based on available job topics and job queue
configuration. • Distributes in the whole topology, including clustered and non-
clustered instances • Can execute cluster local jobs only
![Page 65: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/65.jpg)
65
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
com.adobe.granite.offloading.core
• Builds on top of Sling Distributed Jobs • Does not perform distribution • Detects jobs distributed to non-clustered instances • Transports the jobs and payload to non-clustered
instances • Uses replication for transport • Does not execute jobs
![Page 66: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/66.jpg)
66
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
com.adobe.granite.workflow.core
• Defines a job consumer to distribute the execution of whole workflows
• Defines topic com/adobe/granite/workflow/offloading
• Implements WorkflowOffloadingJobConsumer component
• Supports clustered and non clustered installations
![Page 67: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/67.jpg)
67
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
cq-dam-core
• Makes use of com/adobe/granite/workflow/offloading topic from Workflow Distribution
• New workflow step (external step) that creates a new job on topic com/adobe/granite/workflow/offloading
• New “DAM Update Asset Offloading” workflow • Supports clustered and non clustered
configurations
![Page 68: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/68.jpg)
68
TAKE AWAY Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
Fro
m E
ven
ting
to
Jo
b P
roc
ess
ing
![Page 69: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/69.jpg)
69
TAKE AWAY Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
Fro
m E
ven
ting
to
Jo
b P
roc
ess
ing
New OSGi Specification • Distributed Eventing • Cloud Computing
![Page 70: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/70.jpg)
70
TAKE AWAY Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
Fro
m E
ven
ting
to
Jo
b P
roc
ess
ing
Job Distribution • Improved load balancing • Pull based distribution
![Page 71: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/71.jpg)
71
TAKE AWAY Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
Fro
m E
ven
ting
to
Jo
b P
roc
ess
ing
Offloading • As part of job distribution • Even simpler setup
![Page 72: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/72.jpg)
72
TAKE AWAY Questions?
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
Fro
m E
ven
ting
to
Jo
b P
roc
ess
ing
![Page 73: Adobe AEM - From Eventing to Job Processing](https://reader033.vdocument.in/reader033/viewer/2022050711/554f7991b4c905d25b8b46c9/html5/thumbnails/73.jpg)
73