Why use Azure Functions to Facilitate Data Integration Development Patterns

Working as a Dynamics Consultant for the past seven years has exposed me to a variety of problems to solve. From D365 Finance & Operations (FO) to Customer Engagement (CE) integrations, to using Common Data Model (CDM) or not using CDM, replacing Simple Object Access Protocol SOAP services with Function Apps or Logic Apps, and everything in between with PowerApps replacing the separate line of business applications. No matter the specifics in the scope of work, the root issue was “how do we make business decisions using data that is located outside of our business platform?”

This article will cover exactly where and when Azure Functions can be leveraged within a data integration development pattern. With that understanding, please acknowledge that this topic is vast and growing. We will do our best to keep the focus on where Functions matter, but know we are leaving out design decisions and other variables, which could impact the useful application of what we discuss.

Context

Here is a simple bulleted list providing the structure of the Azure world and where Functions fit within that context. For more information see the Azure Overview documentation.

Azure Subscription – Microsoft Cloud physical segmentation usually by cost center.

- Azure Active Directory (AD) – AD in the cloud, necessary for authentication and authorization.
- App Services Plan – bundles together resources, which will scale together.

- - Azure API Management – provides front facing management tool for an API approach to an integration.
  - Azure Apps

- - - Web Apps – traditional web applications in “as a service” format.
    - Logic Apps – functional focus with pre-built connectors and business logic tools built in.
    - Api Apps – development focus for RESTful api application development.
    - Functions – development focus for complete custom logic and more diverse source code options (PHP, C#, JAVA, Node, etc.)

Common Benefit from Azure Functions

Regardless of the integration or any other purpose to run an Azure Function, there are some traits that obtain common benefits when using them.

1.) Code can be native to existing skill set

No need to learn new languages or platforms or hire consultants/developers

2.) Fits into existing code deployment models through Visual Studio

3.) Legacy code that contains logic can be easily lifted into a Function and leveraged with no need to revisit business requirements

Of course, always revisit requirements for the best outcomes, but time on a project can force this shortcut to occur.

4.) Transform a legacy integration into an API

RESTful endpoints
Event driven
Scheduled or real time

5.) Scalability managed by Azure App Service Plan

Functions can scale based on need
Improves performance
Careful configuration can provide a cost savings

6.) Monitoring through App Insights

App Insights provide analytics on functions OOTB, with options to add additional telemetry code to improve assessment and quality of analytics

7.) Microservices approach

Create functions for individual lines of business, or even more for individual relative pathways of logic.
- For example, instead of a function to handle logic for creating Leads when importing from a marketing source.
  - Create a function for Qualified Leads, Disqualified Leads, for Contacts that are not Leads.
  - Each individual function can be managed by a centralized/orchestration function which calls the other functions or logic apps etc.
Simplified code base and streamlined deployment
- Less risk of changing code for one branch of logic and it impacts another area unintentionally.

Data Integration Patterns

For this discussion, the Data Integration Development Pattern needs a Service Oriented Architecture (SOA), which means something needs to be serving up or opening a service endpoint for access to the data where it lives. The following patterns will assume these services already exist.

1.) ETL (Consolidation)

Scheduled jobs which extract from a source, transforms based on logic, and loads into a target.
Function apps would be the ‘transformation’ portion of this pattern.
- Contains logic to cleanse, normalize or alter relationships
- Can run on schedule

2.) EII – Enterprise Information Integration – Pull/Read Only (Federation)

Just in time data provided for the user where source data is not altered, and target data is not updated.
Function apps would provide an endpoint for business application to trigger the need for data
- Code reaches out to sources and provides a response to target
- Can expose webhooks, monitor other event driven tasks (servicebus), and HTTP API endpoints

3.) Propagation – Updates to several targets

In place when data is needed all the time everywhere
Multiple sources mapped to multiple targets, think Master Data Management
- Function apps would provide schedule or monitoring for event tasks
Could replace legacy middleware that is orchestrating legacy SOAP calls

4.) CDC – Changed Data Capture – Push with ETL

Event driven and near real time
Function app provides logic for transformation if needed
- Flexible trigger or scheduled event for the same logic allows for quick testing or PoC, or hybrid approach with ETL
API Management provides a tool to manage the different functions behind API endpoints
- Central access for management as one API
- Developer portal allows team access to API documentation

5.) Virtualization – Pull/Read only

No need for single data model
Data definitions in target are more flexible
D365 virtual entities would follow this pattern
- Issues:
  - Source must use GUID as PK
  - Cannot create complex relationships
  - Data is not stored in D365 thus no available for reporting
  - Performance hit if used in several locations
Function app would not have the same limitations as D365
- Logic can handle any source to target
- Scalability is automated by Azure App Service Plan

Azure Lifecycle

The following images are from Microsoft and represent the lifecycle for Azure Functions and Azure APIM.

Conclusion

Why use Azure Functions? Not just because they are there.

Use Functions in your integration to:

Breakdown large groupings of logic
- Clean up comments from 2008 and logic so sensitive patches or lifecycle management is a thing of the past.
Improve performance by leveraging durable functions in the consumption model.
Improve monitoring with no extra purchase of monitoring software.
Centralize deployment and management of various apps you may have running.
Create a modern version of legacy integration, may provide cost savings once legacy servers are no longer needed