Common Data Integration Patterns

Migration

Migration is the movement of data from a source system to a destination system at a specific point in time. Data from a source system is filtered using criteria based on the scope of the migration. The data is then transformed to a standard format before it is copied to a destination system. The final state of the replicated data will be inspected to ensure the integrity of the migration.

Broadcast

Broadcast is the movement of data from a source to several destination systems continuously. This data integration pattern is transactional in that the logic executes only for the data that has updated since the previous time it moved.

Bi-directional Synchronization

Bi-directional synchronization is the union of two datasets from two different systems to perform as one while still existing separately. Bi-directional sync ensures that both systems have a consistent, real-time view of the data in both systems. This is valuable to organizations who need to use different applications from different suites simultaneously. This pattern will have a mechanism to identify and correct records modified in both systems in between two synchronization.

Correlation

Correlation is the bi-directional sync performed at the intersection of datasets from two different systems. This pattern does bi-directional sync only with the data that is relevant to both systems. Because irrelevant data is not synchronized, data integration is simpler and more efficient.

Aggregation

Aggregation merges data from multiple systems into one system, providing a centralized view of real-time data from various systems. Aggregation can retrieve, merge, and transform data from multiple sources as needed without requiring an additional combined system. This pattern will have a mechanism to identify and correct records modified in both systems in between two synchronization. Merged records may require human intervention as well.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Arun Singh

Arun Singh

Work as Enterprise Data Architect, Cloud Data Architect and focuses on building data architectures on cloud platforms. www.linkedin.com/in/arun-k-singh-3221372