Azure Integration ServicesKarim Vaes

Specialist – Azure Application Development

@kvaes

Agenda• Digital Transformation

powered by Integration

• Azure Integration Services

APIs API Management

Workflows Logic Apps

Messages Service Bus

Events Event Grid

Why is integration more

important now than ever?

Integration scenarios

Integration challenges

Integration Platform as a Service

Azure Integration Services

APIs Workflows

Messages Events

API Management Logic Apps

Service Bus Event Grid

Azure API ManagementPublish APIs safely and connect to backend systems hosted anywhere

• Work with any host, API, and scale

• Attract more developers

• Secure and optimize your APIs

• Gain insights into your APIs

Azure Logic Apps

Automate workflows and orchestrate business processes easily

• Out-of-the-box connectors reduce integration challenges

• Connect and integrate data from the cloud to on-premises

• B2B and enterprise messaging in the cloud

• Powerful web-based workflow designer

Azure Service Bus

Scalable and reliable cloud messaging as a service

• Simplify enterprise cloud messaging

• Build reliable, scalable cloud solutions

• Implement complex messaging routing

Azure Event Grid

Simplify event-based app development with a publish-subscribe model

• Simple HTTP-based event delivery

• Build better, more reliable applications through reactive programming

• Focus on product innovation

Azure Integration Services

A complete, industry leading integration platform

That is simple to use yet powerful

That is tried and trusted

Integrated and cohesive

Where you need it

Our Vision

API

ManagementLogic Apps Service Bus Event Grid

Integration for the many, not the few

Business

Integration

Specialist

Integration

Developer

Bring your technology investments together

Azure servicesOn-premises or cloud-based apps

Packaged or SaaS apps

© Microsoft Corporation

Learn more about Azure Integration Services

Microsoft Integration Platform as a Service

White paper Introducing Azure Integration Services

Gartner’s Magic Quadrant for Enterprise Integration Platform as a Service

Azure Essentials – Integrating your Apps with Azure

White paper Driving Digital Transformation in Today’s API Economy

Integration Services Customer Stories

Integration Services Reference Architecture

©Microsoft Corporation

Azure

A New Free Whitepaper…

https://aka.ms/integrationpaper

API Economy

“

.”

- .

Incremental Revolution

Digital

Transformation

Agile

Mobile

Machine Learning

Customer Centered Design

Data Driven Decision Making

Automated Marketing

Data Science

DevOps

Cloud

Application Programming Interface (API)

A set of procedures or protocols that allow the data and functionality of an application or service to be accessed and integrated into another

application or service.

What’s in common?

Mobile

Cloud

Computing

Internet

of

Things

Machine

Learning

Software

as a

Service

Blockchain

APIs

APIs: Connecting applications

Benefits of APIs

Lower

TCO

Highly scalable

business models

Foster

innovation

Expand

market reach

Strategic value of APIs

Business models

Channels

Ecosystems

Mobility

User experience

Crowdsourcing

Agility

Empowerment

Productivity

User engagement Ecosystems Multi-modal IT

Based on Gartner research note “Articulating the Business Value of APIs”

Anne Thomas and Kristin R. Moyer

24 March 2016

APIs: Internal architecture for the Enterprise

MAXIMISE IP REUSE ACCELERATE

DEVELOPMENT

CONSISTENCY OF

INFORMATION

CONTROL ACCESS INTRAPRENEURSHIP INSIDE-OUT

TRANSFORMATION

“

.”

- .

©Microsoft Corporation

Azure

The rise of the API Economy

TECHNOLOGY

TRENDS

NEW BUSINESS

STRATEGIES

NEW

REGULATIONS

TECHNOLOGY

STANDARDS

Commercial Models in the API Economy

API Commercial

Models

Developer Pays

Tiered

Pay As You Go

Unit-based

Freemium

Developer Gets

Paid

Ad Revenue-

ShareAffiliate

Cost Per Action Cost Per Click Referal

One-time

Recurring

Indirect

Internal ROIContent

AcquisitionSaaS

Included

Upsell

Traffic

Generation

Who are building successful API businesses?

©Microsoft Corporation

Azure

Risks of exposing APIs

CYBER ATTACKS OVER EXPOSURE OF

ASSETS

REPUTATION

DAMAGE

CANNIBALIZATION

OF CORE BUSINESS

Who is using API Management?

Joining the API Economy

UNDERSTAND THE

VALUE PROPOSITION

DETERMINE

MONETISATION

STRATEGY

ASSESS TECHNICAL

SKILLS

UNDERSTAND DATA

PRIVACY

IMPLICATIONS

MANAGE SECURITY

POLICIES

DEFINE LEGAL

TERMS AND

CONDITIONS

PLAN THE PRODUCT

LIFECYCLE

API Management

API Management

The process of creating and publishing APIs, enforcing their usage policies, controlling access, supporting the subscriber community, collecting and analyzing usage statistics, and reporting on performance.

APIs as products

How do you make your legacy API modern?

How do you protect your core business systems?

How do you enforce your IT and business policies?

How do you engage with developers?

How do you reduce “Time to First Successful Call”?

How do you measure their use and impact?

API Management

Roles of an API management platform

API RegistryFeatures

Structures

Capabilities

Sensitivities

API GatewayManifestation

Security

Entitlement

Standardisation

Logging

Developer PortalLearning

Exploration

Implementation

Feedback

Developer portal

Publisher portal

Gateway

Key Features

Hosted anywhere.

Developed using any

technology.

“

.”

-

Azure API Management

Cloud hosted, turnkey solution

Works with any host, API, and scale

Promotes and supports developer engagement

Secures and optimizes APIs

Provides API insights and analytics

Integration with the Azure ecosystem

©Microsoft Corporation

Azure

API Management - a hub for enterprise APIs

Consume PublishMediate

Azure portalGatewayDeveloper portal

Abstract

Secure & protect

Evolve

Monitor

Analyze

Productize

Monetize

Discover

Learn

On-board

Try

Get support

SDKs and samples

API ManagementDeep Dive

There is a policy for that

Encapsulate common API management functionsAccess control, Protection, Transformation, Caching, …

Mutate request context or change API behaviorAdd a header or throttle for example

Set in the inbound and outbound directions

Applied at a variety of scopes or on errorScope determines which APIs are affected

Can define custom scopes in addition to four available b default

Composed into a pipeline from effective scopesDegree of control over inheritance of scopes, i.e. <base/> element

Don’t delete <base/> inadvertently

Policy scopes

global

product

api

operation

to backend

from backend

from caller

to caller

GET /foo/bar HTTP/1.1

Host: api.constoso.com

Key: 01234567890123456789

/foo

/bar

©Microsoft Corporation

Azure

Policy expressions

C# “snippets” embedded in policy documents

Have read-only access to the request context

Can only use whitelisted .NET types

Dynamically configure and conditionally execute policies

Façade and front door

Developer portal

Azure portal

Gateway

Publish

Mediate

Consume

contosoapi-foo.azurewebsites.com

contosoapi-foo.azurewebsites.comcontosoapi-bar.azurewebsites.com

Façade and front door

Gateway

Mediate

contoso.azure-api.net/fooapi.contoso.com/foo

Security and protection

• Username/Password

• Microsoft account

• Google account

• Facebook account

• Twitter account

• Azure AD (Premium)

• Azure AD B2C (Premium)

• Delegated• Key

• OAuth 2

• OpenID Connect

• Client certificate

• IP filter

• Rate limits and quotas

• Azure account

• RBAC

• HTTP Basic

• Mutual certificate

• Shared secret

• IP filter

• VNET/NSG

Developer portal

Azure portal

Gateway

Publish

Mediate

Consume

VNETs and Hybrid

Developer portal

Azure portal

Gateway

Publish

Mediate

Consume

VNET

VNETs and Hybrid

Gateway

Mediate

VPN

VNET

©Microsoft Corporation

Azure

Multi-region and scaling

27 public regions in Americas, Europe, Asia and Australia

6 US government and DoD regions (preview)

Preview in China!

©Microsoft Corporation

Azure

Multi-region and scaling

©Microsoft Corporation

Azure

Multi-region and scaling

©Microsoft Corporation

Azure

Multi-region and scaling

Versioning is a highly debated subject

Version or not?

Semantic versioning?

What is a breaking change?

Where to place version information?Path? Query? Header? Media type?

What format to use?Number? Date? Name?

Our approach to versioning

Versioning is opt-in

Choose appropriate versioning scheme for an API

Create new API versions as first-class objects

Revise in-use APIs with confidence

Make developers aware of revisions and versions

A note about Swagger (Open API)…

Swagger became OpenAPI Specification

©Microsoft Corporation

Azure

DevOps Integration

https://github.com/Azure/azure-api-management-devops-example

Azure API Management

Cloud hosted, turnkey solution

Works with any host, API, and scale

Promotes and supports developer engagement

Abstracts, protects and optimizes APIs

Provides API governance, insights, and analytics

Stay in touch

Logic Apps

Azure Logic Apps

• Create business processes and workflows visually

• Integrate with SaaS and enterprise applications

• Unlock value from on-premises and cloud applications

• Automate Enterprise Application Integration , Electronic

Data Interchange, and business processes

• Leverage the Microsoft Cloud to enhance integration

solutions

Azure Logic Apps

Fast integration using innovative Visual Designer

Easy workflow creation with triggers and actions

More than 200 connectors to mashup

applications, data and services

No code logic: If, For Each, Do Until, variables and

much more

Built for mission critical 24x7 Enterprise

Integration

DevOps built-in: Create, deploy, manage and

monitor

At the center

Connect on-premises, hybrid

and cloud applications

Run mission-critical, complex

integration scenarios with ease

Connected

Built-in connectivity to 200+ SaaS cloud

services, file providers, databases, web

APIs, productivity apps, and more

Connect to on-premises systems via

Data Gateway

Pluggable extensibility via Custom

Connectors to integrate existing LOB

systems into Logic Apps including SOAP

support

SAP ECC / S/4 HANA connector

• Easy bi-directional SAP integration with Azure

• Send data to SAP ECC and S/4 HANA systems

• Push data from SAP

• Put and Get data from Logic Apps

• IDOC, RFC and BAPI support

• No SAP system footprint

• Generate schemas for mapping

• http://aka.ms/saplogicapps to get started

B2B and EDI Features

• X12

• EDIFACT

• AS2

• RosettaNet (preview)

• Trading Partner Management

• XML processing

• Data Mapping – XSLT 1.0, 2.0, 3.0 and Liquid

Inline CodeJavaScript

• Write code in Logic Apps

• Reference Logic App tokens directly in your

code

• Pass structured results back to Logic Apps

• JavaScript support today

• PowerShell coming soon

• C# and more on roadmap

• http://aka.ms/inlinecode to get started

IBM 3270 Screen Scraping

• Drive mainframe applications via their

screens – “RPA”

• Simple to use Logic Apps connector

• Create mapping and navigation

• 5250, CICS and Host File planned

Developer-centric

• Visual Studio 2019

• Visual Studio Code

• Testability – Static Results

• DevOps

Azure Hybrid Integration Platform

Connect on-premises data and

applications with SaaS, PaaS

and cloud applications

Run mission-critical, complex

integration scenarios with ease

Connect to Azure services with

dozens of Azure connectors

Connect with business partners

Integration Service Environments

• VNET connectivity

• Private static outbound IPs

• Dedicated compute

• Isolated storage

• Higher data transfer rate

• Rules-based Auto-scale

• Flat cost

A complete integration platform

Example - Enterprise integration using message queues and events

https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/enterprise-integration/queues-events

References

Logic Apps Documentation http://aka.ms/logicapps-docs

Logic Apps Blog http://aka.ms/logicappsblog

Logic Apps Monthly Webcast http://aka.ms/logicappslive

Microsoft Integration on Channel 9 https://channel9.msdn.com/Blogs/Microsoft-Integration

Logic Apps Ideas http://aka.ms/logicapps-wish

Logic AppsDeep Dive

Recurrence & Adv. Scheduling

Polling

Webhook

Request

Triggers

An event that starts a workflow instance

Two TypesPolling Trigger

Push Trigger

Not all connectors support triggersOnly specific connectors

Only for specific actions

Recurrence

88

Uses the Schedule – Recurrence connector

Specify how often to start a new instance

Can specify when recurrence should startUseful for controlling time it will run

Polling Triggers

89

Most connector-based triggers use polling

Frequency can be specified

If no events fetched, action still billed

ExamplesService Bus Queues / Topics

Blob Storage

Azure Queues

Event Hubs

Dynamics

HTTP & API Triggers

90

Kinds of Polling Triggers

When frequency is hitCall an external HTTP or API

Start workflow with results

API:API App published on App Service

Request Trigger

91

Push Trigger

Exposes Logic App as an HTTP Endpoint

Uses POST by defaultBut can specify the method

Can receive parameters in URI / Query{parameterName}

Body can be JSONDefine a schema for body

Used to support Nested Workflows

Request / Response

92

If only Request TriggerReturns “202 Accepted” immediately

Can use Response action to:Set HTTP status

Set HTTP headers

Set response body

WebHook Trigger

93

Push-based Trigger

On Save:logic app registers with remote API

Remote API invokes Logic App through

HTTP requests

On Deleted:Logic app unregisters with remote API

WebHook Trigger

94

Debatching

95

Uses “splitOn” the trigger

Alternative to handling array of data in

ForEachStarts a new Workflow Instance for each part

Each instance can be resubmitted

independently

Each instance has the same Correlation ID

Invoke APIs &

Services

Behaviors

Flow Control

Variables

Message Handling

Expressions & Operators

Variables

97

Use “Initialize Variable” to create new one

Supported Data Types:Boolean

String

Integer

Float

Array

Object

Retrieve variable value through

expression:@variable('variableName')

Data Operations

98

Data conversionsGenerate CSV / HTML

Parse JSON

Data ProcessingFiltering arrays

Querying arrays

Join array elements into a string

Author a JSON object

Miscellaneous

99

Expressions

Conditions

Switch Blocks

ForEach

Do Until

Workflow ExpressionsAny input that will be dynamic (change every run) is an expression

@triggerBody() @body(‘actionName’) @items()

@guid() @utcnow() @rand(0, 2)

@json() @xml() @formDataValue(‘actionName’)

@Used to indicate an expression, can be escaped with @@

e.g. @UtcNow()

( )Encapsulate expression parameters

e.g. @length(‘Hello World’)

{ }Curly braces means string. Same as doing @string(), but allows for interpolation

e.g. This is @{add(1,1)} == @concat(‘This is ’, string(add(1,1)))

[ ]Used to parse properties in JSON objects

e.g. @body(‘JsonBody’)[‘person’][‘address’][‘streetName’] 🙂

Conditions

106

IF/Else statement

Condition evaluates to Boolean value[Yes] branch

[No] branch

Is a billable action

Condition

107

By default, simple conditionIs equal to

Is not equal to

Is greater than

Is greater or Equal to

Is less than

Is less than or equal to

Custom expression can be definedOnly in advanced mode

Switch

108

Like a Switch / Case statement

A single input value

Uses equality comparison

[N] case blocks

1 Default Block

ForEach

109

Executes 1..N actions for each item

in an array

Can have 1 or more child actions

Maximum 100,000 items

Pre-filter array using Query action if

necessary

ForEach – Parallel / Sequential

110

Parallel

• Used by Default

• Default of 20

• Maximum 50

Sequential

• Optional

• Enabled in settings

Do Until

111

Runs at least once

1 minute delay after each iteration

Iterates until limits are reachedCondition is true, or

Timeout expires, or

Max iterations are reached (up to 5000)

Used to wait for Async operationsInvoking long running workflow / http

Er

Expressions

Conditions

Switch Blocks

ForEach

Do Until

Retry Policies

113

Automatic retries for any action that failed429 or 5xx error

By default: 4 retries, 20 second interval

Policy can be customized per action

Retry Policy can be disabled (None)

Scopes

114

Groups multiple actions

Can be nested

When an action in a scope fails, execution

can jump to next step after scope

Run After

115

Order in Logic Apps is built out of Run-After chains:Every action specifies after which action it should run

Workflow is not a strict sequence

By default, Run-After is configured to run “on success”

You can specify multiple conditions

Run After - Failure

116

You can change default flow:Run this step only if previous step fails

Allows for compensating actions

Use @result('action name') to get detailed information of errorThis returns an array

Run After - Parallelism

117

Run After can also introduce parallel branches of execution

Can also be used to rejoin parallel branchesSpecify multiple parent actions

Triggers & Run History

Trigger HistoryPopulated when trigger runs

Indicates if any workflows were fired

Inputs & Outputs can be obtained

Run HistoryPopulated when a new workflow instance is created

Visual representation of execution

Can review inputs & outputs for every step

Triggers & Run History

120

Data retained for 90 days

Can only be searched by Instance Identifier

Resubmission

121

Run History allows you to “resubmit” a workflow instance

Resolve issues manually and process messages again

Design workflows to support resubmission

Batching

Completing queues

Often requires using nested logic-apps design

Diagnostics Logs

122

Can be turned on to persist / export diagnostic dataStorage Account (retention can be specified)

Stream to Event Hub

Send to Log Analytics

Two kinds:WorkflowRuntime

Metrics

Using Log Analytics

123

Can connect logic app to OMS workspaceturn on diagnostics

Add the “Logic Apps Management (Preview)” solution to OMS Workspace

Security – Authorization on HTTP endpoints

125

• All calls are done over HTTPS

• Request should be authenticated with Shared Access Signature (SAS)

• HMACSHA256 signed with access key

• Specifies HTTP methods allowed

• Access keys can be rotated

• Invalidates any existing tokens

Security – IP Restrictions

126

Used to control where HTTP calls are allowed fromAny IP

Only other Logic Apps (any subscription)

Specific IP Ranges

IP Restrictions can also be applied to run history APIs.

More complex security:Put Logic Apps behind API Management

Limits – HTTP Requests

127

Limit Value

Request Timeout 120 seconds

Message Size 100MB (depends on connector)

Expression Evaluation Limit 131,072 characters

Retry Attempts 90 (default 4)

Retry Max Delay 1 day

Retry Min Delay 5 seconds

These limits apply to a single HTTP request or connector call

Run Duration and Retention

128

These limits apply to a single workflow run

Limit Value

Run Duration 90 days

Storage Retention 90 days from start time

Min recurrence Interval 1 second

Max recurrence interval 500 days

Looping and Debatching Limits

129

These limits apply to a single workflow run

Limit Value

ForEach items 100,000

Until Iterations 5,000

SplitOn items 100,000

ForEach Parallelism 50 (default 20)

Throughput Limits

130

These limits apply to a single workflow run

Limit Value

Action executions per 5 minutes 100,000

Actions concurrent outgoing calls ~2,500

Runtime Endpoint: Concurrent Incoming Calls ~1,000

Runtime Endpoint: Read calls per 5 minutes 60,000

Runtime Endpoint: Invoke calls per 5 minutes 45,000

Pattern: Claim Check

132

Purpose• Avoid sending large payloads on Logic App Messages

Description• Store payload in Storage Blob

• Send metadata in message to Logic App (say, ServiceBus)

• Include URL of payload in message

Pattern: Nested Workflows

133

Purpose• Support easier resubmission of messages for re-processing in case of failures

Description• Split workflow in two:

• Receiver (receives request, sends reply if needed)

• Processor (processes one message)

• Allows resubmission of Processor instance only

• Receiver workflow can do debatching

• Can re-process individual messages rather than entire batch

Pattern: In-order Processing

134

Purpose• Process events in the order they arrive

Description• Use transport capable of order (like Service Bus Queue)

• Use a singleton workflow

• Make ForEach sequential (remove parallelism)

Pattern: Keep messages in Queue on failures

135

Purpose• Prevent lost messages due to transient failures

Description• Use a peek-lock trigger to receive messages from Service Bus queue

• If processed successfully, complete the message (remove from queue)

• On failure, message eventually goes back to queue to get reprocessed

• Consider moving messages to dead-letter as an option

• Note: Be careful with retries, as lock could expire while waiting for retries to

complete

Pattern: Maximum Throughput

136

Purpose• Maximize processing capabilities in the platform

Description• Increase batch sizes (for triggers that support it)

• Use multiple logic apps sharing a connection or use multiple connections

Logic App Deployment

138

ARM

Resource

Workflow

Definition

Connections

Logic Apps – ARM Resource

139

1 per Logic App

Resource of type Microsoft.Logic/workflows

Defines:Pricing Plan

Location

Workflow definition

Host information

Logic Apps – Workflow Definition

140

JSON document describing workflow

Stored in the definition property of the ARM resource

Usually has parameters for things such as connections

Logic Apps - Connections

141

A connection stores:Metadata for a connector

Securely stores credentials for the connector

Is an ARM resource of type Microsoft.Web/connections

A connection:Can be used by more than one logic app

Must be in same Resource Group as logic app

Exporting Logic App Template

142

Use ArmClient + LogicAppTemplateCreator:https://github.com/jeffhollan/LogicAppTemplateCreator

Exports Logic App + Connectors to new ARM Template

Adds basic parameters

armclient token $subscriptionId | `Get-LogicAppTemplate -LogicApp $appName `

-ResourceGroup $resourceGroup `-SubscriptionId $subscriptionId | `

Out-File template.json

Authorizing connections

143

Some connections can be completely scripted:Example: Service Bus connection resource includes connection stringOAuth-based connections require explicit authorization stepGo into portal and complete authorization processUse https://github.com/logicappsio/LogicAppConnectionAuthOAuth authorization process requires manual intervention

Azure Logic Apps Tools for Visual Studio

144

Create/Edit logic app and get deployable

ARM template

To use:Create new Azure Resource Group project

Add new Logic Apps resource to template

Right click and select “Open with Logic Apps

designer”

Service Bus

Azure Service Bus

Auto-complete

Peek-lock and complete/abandon

Immediate triggering

High throughput

The basics of a queue

Topics & Subscriptions

“type” = “order”

“type” = “quote”

Topic

Subscription 1

Subscription 2

Relay: Real-time communication proxy

Firewall Firewall

How Relay works

Firewall Firewall

Where does Service Bus fit in?

Decoupling and providing durability are primary drivers for using Service Bus

Frontend / UX Backend / Processing

Queue

Service Bus Capabilities

Scheduled delivery

Time to live

ForwardTo

Defer

Sessions

Batching

Ordering

Auto-delete on idle

OnMessage

Duplicate detection

Lambda Filters

Actions

Transactions

Poison message handling

Event Grid

Azure Event Grid

Benefits

PubSub for two different models

Service BusEnterprise Messaging

Event GridReactive Programming

Event Grid: Manage all events in one place

Azure Functions

Logic Apps

Azure Automation

WebHooks

Event Hubs

Storage Queues

Hybrid Connections

IoT Hub

Blob Storage

Resource Groups

Azure Subscriptions

Event Hubs

Custom Topics

Azure Media

Services

Service Bus

Manage all events in one place

Build applications efficiently

Ensure reliability and performance in your apps

Benefit from broad coverage

Scenarios

© Copyright Microsoft Corporation. All rights reserved.