Member-only story
How to use Azure Private Link for Secured Cross-Tenant Data Replication Near Real Time
Credit to my colleague, Alvin on prompting me this simple, yet not straightforward challenge!
Here’s the challenge: IoT data is captured as CSV, and stored in an Azure Storage account in tenant A which is managed by external vendor. Customer operates a data lake based on Azure Data Lake Storage Gen 2 (ADLS) in their own tenant, let’s call it tenant B. Now, the technical requirement is to have a near real time capability to move this IoT data from tenant A, to ADLS in tenant B securely.
Of my head, there are several options available, but upon deeper discussion, these options have pros and cons.
[1] Azure Data Factory
One possible option for transferring files between different Azure tenants is to use Azure Data Factory, a cloud-based data integration service. This option has the advantage of being simple and easy to implement, as Azure Data Factory provides built-in connectors to Azure Blob Storage and Azure Data Lake Storage Gen 2, which are common storage services used by Azure tenants.
Moreover, Azure Data Factory supports managed virtual network, which allows data movement between these tenants to be done securely and privately. However, this option also has a significant…
