Thread 03 – Data Engineering with Fabric

Full versus Incremental Loads The loading of data from a source system to target system has been well documented over the years. My first introduction to an Extract, Transform and Load program was DTS for SQL Server 7.0 in 1998. In a data lake, we have a bronze quality zone that supposed to represent the raw data in a delta file format. This might include versions of the files for auditing. In the silver quality zone, we have a single version of truth. The data is de-duplicated and cleaned up.…

Thread 02 – Data Engineering with Fabric

Managing Files and Folders What is a data lake? It is just a bunch of files organized by folders. Keeping these files organized prevents your data lake from becoming a data swamp. Today, we are going to learn about a python library that can help you. Business Problem Our manager has given us weather data to load into Microsoft Fabric. We need to create folders in the landing zone to organize these files by both full and incremental loads. How can we accomplish this task? Technical Solution This use case…

Thread 01 – Data Engineering with Fabric

Managed Vs Unmanaged Tables Microsoft Fabric was release to the general availability on November 15th, 2024. I will be writing a quick post periodically in 2024 to get you up to speed on how to manipulate data in the lake house using spark. I really like the speed of the starter pools in Microsoft Fabric. A one to ten node pool will be available for consumption in less than 10 seconds. Read all about this new compute from this learn page. Business Problem Our manager has given us weather data…

New England – Microsoft Developers Group

I am enthusiastic about presenting to the New England Microsoft Developers group on June 14th, 2018. As an added benefit, I am looking forward to catching up with my friend and chapter leader, Andy Novick. The meeting venue is at the following address. Microsoft Technology Center 5 Wayside Road Burlington, MA 01803 I hope to see you at the event. Topic: Introduction to Cosmos Database for the C# developer Abstract: Azure Cosmos database is Microsoft’s new globally distributed, multi-model database service. Right now, there is support for five different application…

MONTREAL CANADA – SQL SAT #758

I am traveling to the city of saints to present at SQL Saturday #748 in Montreal, Canada. It will be a good time to meet old friends, make new ones and learn something new. I am really glad that my friend Dennis has decided to go on the road trip with me. It is really boring driving 375+ miles by yourself! I hope you have time to attend this awesome free event on June 2, 2018. I will be talking in a morning session on the following topic.   Topic:…

New York City – SQL SATURDAY #716

I am enthusiastic about presenting at SQL Saturday #716 New York City at the Microsoft office in New York City on May 19, 2018. Of course, I am looking forward to seeing new friends, making new acquaintances, and learning something new. The meeting venue is at the following address. Microsoft Technology Center 11 Times Square New York, NY 10036 Details about the presentations are below. Topic: Standard and Custom Auditing of Azure SQL Database Abstract: The process of classifying a company into an industry segment has been around since the…

Philadelphia – SQL SATURDAY #714

I traveling to the city of brotherly love to present at SQL Saturday #714 in Blue Bell, PA. It will be a good time to meet old friends, make new ones and learn something new. I am really glad that my daughter has decided to go on the road trip with me. It is really boring driving 350+ miles by yourself! I hope you have time to attend this awesome free event on April 21, 2018. I will be talking in a morning session on the following topic.   Topic:…

PROVIDENCE USER GROUP – April 2018

I am enthusiastic about presenting to the Providence SQL Server User Group on April 18, 2018. Of course, I am looking forward to talking to my friends Matt Batalon and Gregory Celentano. If you have not figured it out yet, I love to chat to local user groups about any SQL server topic that I am passionate about. The meeting venue is at the following address. New England Institute of Technology 1408 Division Rd East Greenwich, RI Details about the presentation are below. Topic: Introduction to Azure Table Storage Abstract:…

Boston – Code Camp #29

I am excited about the opportunity to present at Code Camp 29 in the surrounding Boston area. It will be a good time to meet old friends and make new ones. I hope you have time to attend this awesome free event on November 18, 2017 at the Microsoft Office, Five Wayside Road, Burlington, MA.   Here are the details behind the first presentation that I gave that day. Topic: Staging data for Azure SQL Services Abstract: Most companies are faced with the ever-growing big data problem. It is estimated…

ROCHESTER – SQL SATURDAY #723

Time to travel to the Flour City to present at SQL Saturday #723 in Rochester, NY. It will be a good time to meet old friends, make new ones and learn something new. I am really glad that my daughter Josie decided to go on the road trip with me. It is really boring driving 350+ miles by yourself! I hope you have time to attend this awesome free event on March 24, 2018. Here are the details behind the two presentations that I am giving that day.   First…