Creating a Data Strategy Big data revolution has sparked a trend among organizations to collect all possible data. Cheap storage and processing capabilities offered by hyperscale's of today have ignited this trend further, which has resulted abundance of data. Everyone is capturing and storing this data in multiple ways and “Data Lake” is one…

Data exchange between applications is a key requirement. An organization will typically have a large set of applications in its IT landscape and will require them exchange data on regular basis to accomplish business objectives. A data integration pattern is a standardized method for integrating data. …

A list of design patterns, one must keep in arsenal :) 1. Strategy: Defines a family of algorithms, encapsulates each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients who use it. 2. Decorator: Attach additional responsibilities to an object dynamically. …

Growth of data creation is unprecedented in current times and is expected to grow explode in near future. Ever since storage and data processing/computation capabilities have enhanced, it led to capturing every bit of data. It is available for analysis and deliver insight within. https://www.statista.com/chart/17727/global-data-creation-forecasts/ As per “statista”, data creation…

Apache NiFi Overview Apache NiFi is a dataflow system based on the concepts of flow-based programming. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. NiFi has a web-based user interface for design, control, feedback, and monitoring of dataflows. It is highly configurable along…

What is an incremental model? A common need in building models in data warehouse is to build models/tables incrementally and not truncate/load or drop/load every time. This is an effective strategy to build tables with optimized resource consumption. When building Incremental models as tables in your data warehouse — the…

A data pipeline is a series of data processing steps. If the data is not currently loaded into the data platform, then it is ingested at the beginning of the pipeline. Then there are a series of steps in which each step delivers an output that is the input to…

Extracting Data from SAP DSO in Azure Data Factory There are many types of data stores available in SAP. DSO is one such data stores and has a unique feature to maintain data. DataStore Object (DSO) is used to store data at the most granular form, i.e. at document level. …