Engenharia de dados, proposta de um pipeline ETL para integração de dados das vendas na CDM: caso de estudo - Cervejas de Moçambique

Abstract

Currently, it is irrefutable that the biggest asset in companies is data. Companies deal with competitiveness and to gain competitive advantages, they design strategies and make data-driven decisions. Lately, the term Business intelligence has been gaining notoriety in organizations to drive business intelligently. Organizations have generated thousands of data daily in their operations, the main ones being sales and interaction with customers. This data is generated in different sources and formats, which makes it challenging to use them for analysis and subsequent decision making. Data engineering solutions help to integrate this data, with the concept of data pipeline as they offer mechanisms for centralizing and standardizing data. The main objective of the work is to propose a data pipeline to collect, transform and make data available to generate reports and dashboards. Once this scenario and the consequent constraints were verified, bibliographical research was carried out on the main data integration solutions for an in-depth study of them. To choose the ideal solution for the case under study, it was imperative to carry out a comparative analysis of the possible tools. Based on this comparison, it was concluded that the solution that best suits the CDM company's reality is Azure Data Factory and implementation continued. To test the adopted solution, it was necessary to create a data pipeline reflecting the reality of the business.