Connecting multiple systems requires an infrastructure that moves data between the systems. However, you often want the solution to do more than just pass data around. You want to add an additional layer of functionality on top of the functional assets that reside inside the existing applications. This layer can automate complex business processes or provide unified access to information that is currently scattered across many systems. How should such an integrating layer be designed, and what choices do you have? Unfortunately, there is no single right answer for all enterprise architectures.
In general, when you’re trying to choose an integration pattern, we recommend that you consider the following questions:
- Is there a business requirement that the integration should be in real time?
- What is the requirement for the peak data volume?
- What is the frequency?
Migration Pattern
Migration is the act of moving a specific set of data at a point in time from one system to the other. A migration contains a source system where the data resides at prior to execution, a criteria which determines the scope of the data to be migrated, a transformation that the data set will go through, a destination system where the data will be inserted and an ability to capture the results of the migration to know the final state vs the desired state.
Why is it valuable?
Migrations are essential to all data systems and are used extensively in any organization that has data operations. We spend a lot of time creating and maintaining data, and migration is key to keep that data agnostic from the tools that we use to create it, view it, and manage it. Without migration, we would be forced to lose all the data that we have amassed any time that we want to change tools, and this would cripple our ability to be productive in the digital world.
When is it useful?
Migrations will most commonly occur whenever you are moving from one system to another, moving from an instance of a system to another or newer instance of that system, spinning up a new system that extends your current infrastructure, backing up a dataset, adding nodes to database clusters, replacing database hardware, consolidating systems and many more.
Broadcast Pattern
Broadcast can also be called “one-way sync from one to many”, and it is the act of moving data from Dynamics 365 to multiple destinations in real time.
Whenever there is a need to keep our data up to data between multiple systems, across time, you will need either a broadcast, bi-directional sync, or correlation pattern. The distinction here is that the broadcast pattern, like the migration pattern, only moves data in one direction, from the source to the destination. Now, I know what you are going to ask next, “What is the difference between the broadcast pattern and the migration pattern which is set to automatically run every few seconds?” The main distinction to keep in mind is that the broadcast pattern is transactional meaning that it does not execute the logic of the message processors for all items which are in scope, rather it does it only for those items that have recently changed. So you can think of broadcast as a sliding window that only captures those items which have field values that have changed since the last time the broadcast ran. Another major difference is in how the implementation of the pattern is designed. Migration will be tuned to handle large volumes of data and process many records in parallel and to have a graceful failure case. Broadcast patterns are optimized for processing the records as quickly as possible and being highly reliable to avoid losing critical data in transit as they are usually employed with low human oversight in mission-critical applications.
Aggregation Pattern
Collects data from multiple systems and copies into Dynamics 365 by aggregating data in one process. This will remove the need for multiple one way integrations between source and data systems.
A critical aspect of the aggregation process is managing that into a single entity into Dynamics 365.
Why is it valuable?
The aggregation pattern allows you to extract and process data from multiple systems in one application. This means that the data is up to date at the time that you need it, does not get replicated, and can be processed/merged to produce the dataset you want.
When is it useful?
The aggregation pattern is useful for creating reports or dashboards which similarly have to pull data from multiple systems and create an experience with that data. It is also useful for when you have systems that you use for compliance or auditing purposes which need to have related data from multiple systems. The aggregation pattern can be very helpful here in making it so your data lives in one system, but can be the amalgamation of data from multiple systems. This way you can reduce the amount of learning that needs to take place across the various systems to ensure you have visibility into what is going on.
Bi-directional Synchronization
Bi-directional sync is the act of joining two datasets in two different systems to behave as one while respecting their need to exist as different datasets. The main driver for this type of integration need comes from having different tools or different systems for accomplishing different functions on the same data set. For example, you may have a system for taking and managing orders and a different system for customer support. For one reason or another, you find that these two systems are best and are important to use them rather than a suite which supports both functions and has a shared database. Using bi-directional sync to share the dataset will enable you to use both systems while maintaining a consistent real-time view of the data in both systems.
When is it useful?
The need, or demand, for a bi-directional sync integration, is synonymous with wanting object representations of reality to be reliable and consistent. For example, if you want one view of your customer, you can solve that manually by giving everyone access to all the systems that have a representation of the notion of a customer. With that solution, you can achieve one view of the customer at the expense of productivity, training, security, and cost. A more elegant solution to the same problem is to list out which fields need to be visible for that customer object in which systems, and which systems are the owners of those. Most enterprise systems have a way to extend objects such that you can modify the customer object data structure to include those fields. Then, you can create integration applications either as a point to point applications (using a common integration platform) if it’s a simple solution, or a more advanced routing system like a pub/sub or queue routing model if there are multiple systems at play. With this approach, you can achieve a state where each person who deals with customers can have their own view of the customer with only the information that is relevant to them. For example, a salesperson should know the status of a delivery, but they don’t need to know which warehouse the delivery is at. Similarly, the delivery person needs to know the name of the customer that the delivery is for without needing to know how much the customer paid for it. Bi-directional synchronization allows both of those people to have a real-time view of the same customer from the perspective that they care about.
Correlation Pattern
The correlation pattern is a design that identifies the intersection of two data sets and does a bi-directional synchronization of that dataset only if that item occurs in both systems naturally. Similar to how the bi-directional pattern synchronizes the union of the scoped dataset, correlation synchronizes the intersection.
Why is it valuable?
Correlation is useful in the case where you have two groups or systems that want to share data only if they both have a record representing the same item/person in reality. For example, let’s say you are a hospital group with two hospitals in the same city. You may think something like “Hey, let’s share data between the two hospitals, so if a patient uses either hospital, we will have an up to date record of what treatment they received at both hospitals.” To accomplish an integration like this, you may decide to create two broadcast pattern integrations, one from Hospital A to Hospital B, and one from Hospital B to Hospital A. This will ensure that the data is synchronized, however, you will have two integration applications to manage. To alleviate the need to manage two applications, you can just use the bi-directional synchronization pattern between Hospital A and B. But then you may think “Man, I wish there was a way to only bring the patient record from the other hospital if I create a record for the same person in this hospital.” Meaning that the synchronization should not bring the records of patients of Hospital B if those patients have no association with Hospital A and it should bring it in real-time as soon as the patient’s record is created. The correlation pattern is valuable because it only bi-directionally synchronizes the objects on a “Need to know” basis rather than always moving the full scope of the dataset in both directions.
When is it useful?
The correlation pattern is most useful when having the extra data is more costly than beneficial because it allows you to scope out the “unnecessary” data. For example, if you are a university, part of a larger university system, and you are looking to generate reports across your students. You probably don’t want a bunch of students in those reports that never attended your university. But you may want to include the units that those students completed at other universities in your university system. Here, the correlation pattern would save you a lot of effort either on the integration or the report generation side because it would allow you to synchronize only the information for the students that attended both universities. Or if you are a sales organization, you may want to synchronize data about accounts with other organizations, but you only want to do it for accounts that you both have, and you don’t want to send other accounts to each other. You could not do this automatically without knowing which accounts the other organization has without employing a similar pattern of thought.
