Overcome Azure Data Factory Web Services Limit— Move Huge Volume of Data from Web Services to Databases
--
Azure Data Factory (ADF) is a serverless, cloud-based ETL (Extract, Transform, Load) tool, commonly used as a data pipeline to move data from data sources to other data destinations, be it cloud or on-premises. It provides a graphic user interface, comes with a pre-built connector for both 1st party and 3rd party databases, which make the creation of the data pipeline experience simple and seamless. The best thing? Git versioning is natively built-in, which means that we can bring DevOps practices into our data pipeline. More automation and less human error!
This article is not meant to be Azure Data Factory introduction, there are plenty of valuable resources online, and I would suggest looking at the official documentation: https://docs.microsoft.com/en-us/azure/data-factory/introduction
Situation
As a cloud solution architect, my role is to help my customers to take advantage of cloud platforms, including recommending the right SKU for their computational requirements. One common scenario is to recommend Azure VM sizes. Azure operates in more than 60 regions, each region may have different offerings, in addition to the wide range of offerings like A series, B series, D series, and more. The permutation added up, to more than 15,000-line items! I have goldfish’s memory, not being able to remember all configurations. To make the situation worse, Azure rolls out new instances frequently!
I used to be doing this: Launch Azure Calculator, select location and look through the list of instances (with configuration).
It’s doable for small workloads, but increasingly more organizations move workloads to the cloud, the size of the workload increases too. It’s time-consuming. So, I thought of having a place for me to do filtering, not just on region, but also filter based on CPUs and RAM. I can just drag my filter to the range I want, and voila, I have the SKU filtered out.
Of course, I can write an application to handle this, but the time invested to build…
