I am currently preparing to take the Microsoft Dynamics 365 + Power Platform Solution Architect certification. Or “MB 600” for short! As I prepare I plan to write revision notes in this post I will cover my revision connected with designing integrations.
Note: Microsoft have recently updated some of their terminology. Therefore we should consider the terms like CDS and Dataverse as interchangeable. In terms of the exam I would expect them to begin using the term Dataverse at some point but that change is unlikely to be immediate. Meaning in the short term either term could be used. I created this blog post before the revised terms were announced. I have tried to apply updates but a few “outdated” references may still exist!
The Solution Architect will be required to identify when integrations maybe required. They will lead the design of any integrations and document how they will be included into the overall architecture. It will be the Architect’s role to ensure any integrations do not make the solution “fragile” and may additionally need to consider integrations as part of an overall disaster recovery plan.
There are many reasons we need integrations. Integrations maybe required to provide a consistent interface for users / customers. Maybe real-time integrations are needed to keep disparate systems up-to-date. Often integrations may be required to avoid reinventing the wheel! Reusing an existing system or component with the help of integrations maybe be more cost effective and achievable than reimplementing.
A Power Platform Solution Architect must have a deep technical knowledge of Power Platform and Dynamics 365 apps plus at least a basic knowledge of related Microsoft cloud solutions and other third-party technologies. When designing integrations the Solution Architect will review the requirements and identify which parts can leverage a Dynamics 365 apps and which parts must be custom built by using the Power Platform, Microsoft Azure etc.
Historically an architect might have started with custom development in mind but a Power Platform Solution Architect will typically start their design process with a focus on Dynamics 365 and the Power Platform and only later use Microsoft Azure and other custom approaches to address any gaps.
When considering the design of integrations it maybe worth thinking about how the Dataverse operates. The Dataverse is a software-as-a-service (SaaS) platform, therefore most of its architecture, such as underlying data storage, are abstracted from developers so they can focus on other tasks such as building custom business logic and integrating with other applications.
Historically we may have used SOAP API to programmatically retrieve data from the Dataverse (CDS). But these days the web API is the preferred approach. If you aren’t already familiar with the web API approach you can read about about it here.
Most solutions do not exist in isolation, they rely on internal and external integrations. A part of identifying solution components the Architect should highlight how any integrations will be handled. We may need to define what tools or services will be used to complete the integrations. And also potentially define clear boundaries on where one system ends and another begins. The Solution Architect should focus on what happens on these boundaries. Often multiple parties will be involved in the development of integrations so clearly defining which boundaries are “owned” by which supplier or internal development teams maybe critical.
Integrations can take many forms.
Data integration – “just” combining data from different sources. Maybe to provide the user with a unified view. Data integrations maybe event base (near real-time) or batch based (maybe with overnight updates).
Application integration – a higher level integration connecting at the application layer.
Process integration – potentially you retain multiple disparate systems but each of those systems remain part of the overall business function.
Design collaboration integration
Often integration may involve collaboration tools such as Outlook, Teams, Yammer and more.
SharePoint for example maybe leveraged for document storage, whist Teams can be created to help groups of users collaborate.
Often the associated collaboration tools will sit outside of the Power Platform and Dynamics 365.Meaning each tool would have its own requirements / constraints in terms to license costs, security considerations and more.
Design Dynamics 365 integration
There are a number of different types of integration / extensibility options available to us when considering the integration of Dynamics 365 applications.
Additionally developers can access the transactional pipeline with plugins. (using .net SDK) Or custom workflow activities. As mentioned we can access the Dataverse (CDS) API service using SOAP or web API (OData).
Custom business logic can be created via plugins. This custom logic can be executed synchronously or asynchronously. The results of synchronous calls can be seen by the users immediately but this does mean the user is “blocked” until the call completes. Whilst with asynchronous calls will not block the user. However asynchronous calls can only run post operation. (Synchronous calls can run post or pre-operation.)
Synchronous calls are included in the original transaction but are limited to 2 minutes. Whilst asynchronous customizations would not be part of the original transaction.
FYI: When you use the Dataverse connector in Power Automate or a Canvas App it will be making calls to the OData API. (Power Automate runs asynchronously.) Classic workflows can run synchronously (real-time) or asynchronously.
Whilst server side customizations only happen when the data is sent to the server. Meaning users would only see the results when a data refresh is completed. Server side customizations can be created with plug-ins, Power Automate flows, classic workflows and business rules (with entity scope).
To ensure consistent availability and performance for everyone the platform applies limits to how APIs are used by the Common Data Service. Service protection API limits help ensure that users running applications cannot interfere with each other based on resource constraints. These limits should not affect normal users of the platform. You can read an overview of API limits here.
API requests within the Power Platform consist of various actions which a user makes. Including;
- Connectors – API requests from connectors in Power Apps and Power Automate
- Power Automate – all power automate step actions result in API requests
- Dataverse (Common Data Service) – all create, read, update and delete (CRUD) operations. Plus “special” operations like share and assign.
You can read more about APIs and their limits here. Because limits exist the Solution Architect may need to optimize integrations to minimize API calls. Additionally data integration applications may need to handle API limit errors, this is done by implementing a strategy to retry operations if an API limit error is received. You can read about the service protection and API limits here.
Design internal system integration
Internal app builders / customizers maybe able to leverage low code approaches to support internal integrations. Maybe these integrations can be created using canvas apps or power automate flows with standard or custom connectors. Additionally canvas apps maybe embedded into model-driven apps to leverage connectors.
Power Automate maybe used for event based data integrations especially if integrations are needed between multiple Dataverse environments.
Design third-party integration
Developers maybe required to assist with 3rd party integrations. Maybe this will involve creating custom APIs, developing plug-ins or using other technologies such as Azure Logic Apps or Azure Service Bus.
Virtual entities may also give visibility of external data sources directly in model driven apps. Or maybe custom UI controls can be created to surface 3rd party data using the PowerApps Component Framework (PCF).
With inbound data integrations 3rd party tools such as KingswaySoft or Scribe maybe used. When using such tools performance / throughput may need to be considered, you possibly need to design multiple threads to overcome latency effects.
Web application integrations maybe created using webhooks. Where custom call backs maybe triggered using JSON in a http post request. Webhooks maybe synchronous or asynchronous.
Design authentication strategy
The systems into which integrations are required may demand differing approaches to authentication.
When working in a batch mode for updating data the timing of authentication requests may also be a consideration which could impact performance. Maybe you shouldn’t authenticate on every request!
OAuth is a standard that apps can use to provide client applications with “secure delegated access”. OAuth works over HTTPS and authorizes devices, APIs, servers, and applications with access tokens rather than credentials. Authentication integration maybe provided with OAuth providers such as Azure Active Directory, Facebook, Google, Twitter and Microsoft accounts. With basic authentication the user would always have to provide a username and password. With OAuth the user sends an API key ID and secret.
Design business continuity strategy
In terms of availability and disaster recovery the Power Platform components should handle concerns with internal integrations. Therefore the Solution Architect may need to focus on external integrations. Additionally manual environment backup / restore operations can be used to help with “self-created” problems like bad deployments or mass data corruptions.
Separation of applications can reduce their reliance on each other. Using Azure Service Bus may allow integrations between systems in a decoupled manner.
Design integrations with Microsoft Azure
Webhooks can only scale when the host application can handle the volume. Azure Service Bus and Azure Event Hubs maybe used for high scale processing / queuing of requests. Although the Azure approach can only be asynchronous. (Webhooks can be synchronous or asynchronous.)
Azure Service Bus
CDS supports integration with Azure Service Bus. Developers can register plug-ins with Common Data Service that can pass runtime message data, known as the execution context, to one or more Azure solutions in the cloud. AzureAzure Service Bus integrations provide a secure and reliable communication channel between the common data service runtime data and external cloud-based line-of-business applications. You can read more about Azure integration here.
Azure Service Bus distributes messages to multiple independent backend systems, decoupling the applications.
Azure Service Bus can protect the application from temporary peaks.
Azure Event Hubs
Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters.
The following scenarios are some of the scenarios where you can use Event Hubs:
- Anomaly detection (fraud/outliers)
- Application logging
- Analytics pipelines, such as clickstreams
- Live dashboarding
- Archiving data
- Transaction processing
- User telemetry processing
- Device telemetry streaming
Event Hubs provides a distributed stream processing platform with low latency and seamless integration, with data and analytics services inside and outside Azure to build your complete big data pipeline.
Event Hubs represents the “front door” for an event pipeline, often called an event ingestor in solution architectures. An event ingestor is a component or service that sits between event publishers and event consumers to decouple the production of an event stream from the consumption of those events. Event Hubs provides a unified streaming platform with time retention buffer, decoupling event producers from event consumers.
Azure Logic Apps
Azure Logic Apps is a cloud service that helps you schedule, automate, and orchestrate tasks, business processes, and workflows when you need to integrate apps, data, systems, and services across enterprises or organizations. Logic Apps simplifies how you design and build scalable solutions for app integration, data integration, system integration, enterprise application integration (EAI), and business-to-business (B2B) communication, whether in the cloud, on premises, or both.
Every logic app workflow starts with a trigger, which fires when a specific event happens, or when new available data meets specific criteria. Many triggers provided by the connectors in Logic Apps include basic scheduling capabilities so that you can set up how regularly your workloads run.
In many ways Azure Logic Apps and Power Automate Flows have a lot in common. However; Power Automate Flows can be packaged as part of a CDS solution. And Power Automate CDS connector has more capabilities. Plus Power Automate allows UI Automation.
Azure Functions allows you to run small pieces of code (called “functions”) without worrying about application infrastructure. With Azure Functions, the cloud infrastructure provides all the up-to-date servers you need to keep your application running at scale.
A function is “triggered” by a specific type of event. Supported triggers include responding to changes in data, responding to messages, running on a schedule, or as the result of an HTTP request.
While you can always code directly against a myriad of services, integrating with other services is streamlined by using bindings. Bindings give you declarative access to a wide variety of Azure and third-party services.
Azure functions are serverless applications.
Choosing between Power Automate, Azure Logic Apps, Azure Functions and Azure App Service Webjobs maybe confusing! As all of these solve integration problems and automate business processes. This link may help you compare these options!