Call Graph API from Azure Data Factory

Easy way to copy data from API to db using Azure data factory. In this exaple graphql end point used  to receive API data. 

Azure, Data Factory

Azure data factory is an ETL and orchestrator tool for building cloud-native data engineering pipelines. It has a lot of source connectors available and this list is growing rapidly. Microsoft has also enabled support for accessing Microsoft Graph API from Azure Data Factory. In this article, we will see how to call Graph API from Azure Data Factory using the managed identity.

What is Microsoft Graph API?

Microsoft Graph is a RESTful web API that enables you to access Microsoft Cloud service resources like Users, Groups, Mail, Calendars, Contacts, Files, etc. More information is available on the Microsoft docs.

Authentication

There are two ways to authenticate Graph API from Azure Data Factory.

Using Service principal

The service principle in simple terms is a service account. This enables programmatic access to resources in Azure. To create a service principal, we need to register an App in Azure Active Directory. This application has clientID, client Secret and app ID. These details can also be used to grant access to Microsoft Graph API.

Using Managed Identity

Managed identity is a service principal associated with resources in Azure. When we create a data factory in Azure, it automatically creates an app in Azure Active Directory. This means if we want to access Graph API using the data factory, we just need to grant Azure Data Factory app access to Graph API. This makes access management more secure and easy, isn’t it?

Let’s see it in action

This section describes how to call Graph API from Azure Data Factory using the managed identity. For demo purposes, we will get Azure active directory users’ data using Graph API and copy it to blob storage.

1. Providing Graph API access to Azure data factory.

a. Navigate to Azure active directory on Azure portal and search the data factory application (managed identity) under Enterprise applications.

b. Grant access to Data Factory app on Graph API as shown in the below image.

Note: Granting access to Graph API requires Azure active directory admin consent.

Azure data factory requires “user.read.all” permission to read users’ data from Graph API.

2. Setting up linked service for API calls.

a. Connect to the Azure portal and open the data factory. Create a new pipeline.

b. Create a new Linked service for REST API. Provide Authentication Type and AAD resource values as mentioned below.

Authentication Type : System Assigned Managed Identity

AAD resource : https://graph.microsoft.com/

3. Configure Copy Activity to get data from API and push it to blob Storage.

URL : https://graph.microsoft.com/v1.0/users

Pagination Rules : AbsolutionUrl = ['@odata.nextLink']

Execution of this ADF pipeline should generate a file in the target blob storage.

Pro tips:
1. Microsoft Graph API provides various API filtering parameters. More details can be found at this link.