fbpx

Why use Azure Functions to Facilitate Data Integration Development Patterns

Working as a Dynamics Consultant for the past seven years has exposed me to a variety of problems to solve. From D365 Finance & Operations (FO) to Customer Engagement (CE) integrationsto using Common Data Model (CDM) or not using CDM, replacing Simple Object Access Protocol SOAP services with Function Apps or Logic Apps, and everything in between with PowerApps replacing separate line of business applications. No matter the specifics in the scope of work, the root issue was “how do we make business decisions using data that is located outside of our business platform?” 

This article will cover exactly where and when Azure Functions can be leveraged within a data integration development pattern. With that understanding, please acknowledge that this topic is vast and growing. We will do our best to keep the focus on where Functions matter, but know we are leaving out design decisions and other variables, which could impact the useful application of what we discuss. 

Context 

Here is a simple bulleted list providing the structure of the Azure world and where Functions fit within that context. For more information see the Azure Overview documentation 

  • Azure Subscription – Microsoft Cloud physical segmentation usually by cost center. 
    • Azure Active Directory (AD– AD in the cloud, necessary for authentication and authorization. 
    • App Services Plan – bundles together resources, which will scale together. 
      • Azure API Management – provides front facing management tool for an API approach to an integration. 
      • Azure Apps 
        • Web Apps – traditional web applications in “as a service” format. 
        • Logic Apps  functional focus with pre-built connectors and business logic tools built in.  
        • Api Apps – development focus for RESTful api application development. 
        • Functions – development focus for complete custom logic and more diverse source code options (PHP, C#, JAVA, Node, etc.)

Common Benefit from Azure Functions 

Regardless of the integration or any other purpose to run an Azure Functionthere are some traits that obtain common benefits when using them.  

1.) Code can be native to existing skill set

  • No need to learn new languages or platforms or hire consultants/developers 

2.) Fits into existing code deployment models through Visual Studio 

3.) Legacy code that contains logic can be easily lifted into a Function and leveraged with no need to revisit business requirements 

  • Of course, always revisit requirements for the best outcomes, but time on a project can force this shortcut to occur. 

4.) Transform a legacy integration into an API 

  • RESTful endpoints 
  • Event driven 
  • Scheduled or real time 

5.) Scalability managed by Azure App Service Plan 

  • Functions can scale based on need 
  • Improves performance 
  • Careful configuration can provide a cost savings 

6.) Monitoring through App Insights 

  • App Insights provide analytics on functions OOTB, with options to add additional telemetry code to improve assessment and quality of analytics 

7.) Microservices approach 

  • Create functions for individual lines of business, or even more for individual relative pathways of logic. 
    • For example, instead of a function to handle logic for creating Leads when importing from a marketing source. 
      • Create a function for Qualified Leads, Disqualified Leads, for Contacts that are not Leads. 
      • Each individual function can be managed by a centralized/orchestration function which calls the other functions or logic apps etc.  
  • Simplified code base and streamlined deployment 
    • Less risk of changing code for one branch of logic and it impacts another area unintentionally.  

Data Integration Patterns 

For this discussion, the Data Integration Development Pattern needs a Service Oriented Architecture (SOA), which means something needs to be serving up or opening a service endpoint for access to the data where it lives. The following patterns will assume these services already exist.  

1.) ETL (Consolidation) 

  • Scheduled jobs which extract from a source, transforms based on logic, and loads into a target.  
  • Function apps would be the ‘transformation’ portion of this pattern.  
    • Contains logic to cleanse, normalize or alter relationships 
    • Can run on schedule 

2.) EII – Enterprise Information Integration – Pull/Read Only (Federation) 

  • Just in time data provided for the user where source data is not altered, and target data is not updated. 
  • Function apps would provide an endpoint for business application to trigger the need for data 
    • Code reaches out to sources and provides a response to target 
    • Can expose webhooks, monitor other event driven tasks (servicebus), and HTTP API endpoints 

3.) Propagation – Updates to several targets 

  • In place when data is needed all the time everywhere 
  • Multiple sources mapped to multiple targets, think Master Data Management 
    • Function apps would provide schedule or monitoring for event tasks 
  • Could replace legacy middleware that is orchestrating legacy SOAP calls 

4.) CDC – Changed Data Capture – Push with ETL  

  • Event driven and near real time 
  • Function app provides logic for transformation if needed 
    • Flexible trigger or scheduled event for the same logic allows for quick testing or PoC, or hybrid approach with ETL 
  • API Management provides a tool to manage the different functions behind API endpoints 
    • Central access for management as one API 
    • Developer portal allows team access to API documentation 

5.) Virtualization  Pull/Read only 

  • No need for single data model 
  • Data definitions in target are more flexible  
  • D365 virtual entities would follow this pattern 
    • Issues 
      • Source must use GUID as PK 
      • Cannot create complex relationships  
      • Data is not stored in D365 thus no available for reporting 
      • Performance hit if used in several locations 
  • Function app would not have the same limitations as D365 
    • Logic can handle any source to target 
    • Scalability is automated by Azure App Service Plan 

Azure Lifecycle 

The following images are from Microsoft and represent the lifecycle for Azure Functions and Azure APIM.   

Figure 1: Azure Functions Deployment Model
Figure 2: APIM Deployment Model

Conclusion 

Why use Azure Functions? Not just because they are there.  

Use Functions in your integration to: 

  • Breakdown large groupings of logic  
    • Clean up comments from 2008 and logic so sensitive patches or lifecycle management is a thing of the past.  
  • Improve performance by leveraging durable functions in the consumption model. 
  • Improve monitoring with no extra purchase of monitoring software.  
  • Centralize deployment and management of various apps you may have running. 
  • Create a modern version of legacy integration, may provide cost savings once legacy servers are no longer needed 

In the cloud-focused world, we believe Functions can be your first step, even though we know Functions are not the only solution 

Resources 

Azure Functions Overview – https://docs.microsoft.com/en-us/azure/azure-functions/functions-overview  

Figure 1 and 2 from Microsoft’s Chris Anderson Azure Product Team: Leveraging Azure Functions in Azure App Service Integration Scenarios – https://www.biztalk360.com/integrate-2016-resources/leveraging-azure-functions-azure-app-service-integration-scenarios/  

Leave a Reply

You must be logged in to post a comment.