Data Presentation Layer Microsoft Fabric allows the developer to create delta tables in the lake house. The bronze tables contain multiple versions of the truth and the silver tables a cleaned up, single version of the truth. How can we combine the silver tables into a relational model for consumption from the gold layer? Business Problem Our manager at adventure works has asked us to use a metadata driven solution to ingest CSV files from external storage into a Microsoft Fabric. A typical medallion architecture will be used in the…
Tag: John F. Miner III • Microsoft Fabric • Data Engineering • Apache Spark • Data Factory • Full Load Code • Meta Data Driven Design • Learn • CSV files
Thread 04 – Data Engineering with Fabric
Metadata Driven Pipelines What is a metadata driven pipeline? Wikipedia defines metadata as “data that provides information about other data”. As a developer, we can create a non parameterized pipeline and/or notebook to solve a business problem. However, if we have to solve the same problem a hundred times, the amount of code can get unwieldly. A better way to solve this problem is to store metadata in the delta lake. This data will drive how the Azure Data Factory and Spark Notebooks execute. Business Problem Our manager has asked…