Azure Data Factory Tutorial: Your first steps to easy data integration in the cloud

In the modern cloud world, moving and transforming data is an everyday necessity. Whether you need to load data from multiple sources into a central data warehouse, synchronize databases, or prepare data for analytics workloads, a reliable data integration tool is essential. This is where Azure Data Factory (ADF) comes into play, Microsoft's powerful and scalable cloud ETL and data integration service.

Are you new to ADF and wondering how to get started? No worries! This Azure Data Factory Tutorial provides you with a practical ADF introduction and walks you step-by-step through creating your very first data pipeline. By the end, you will understand how to automate simple data copy tasks in Azure.

Requirement: You need an active Azure subscription. If you don't have one yet, you can often start with a free trial account.

What is Azure Data Factory (Quick Overview)?

Azure Data Factory is a fully managed, serverless data integration service. Think of it as an orchestration platform that allows you to create, schedule, and monitor data flows (called pipelines). These pipelines can retrieve, transform, and load data from a variety of sources (on-premises or in the cloud) to various destinations.

The core components of ADF are:

• Pipelines: A logical group of activities that work together to complete a task.
• Activities: Individual processing steps in a pipeline (e.g. copy data, execute stored procedure, execute data flow).
• Linked Services: Define the connection information to external resources (e.g. databases, file storage, cloud services). Like a connection string.
• Datasets: Represent the structure of the data within the data storage (e.g. a specific table, a file, a folder).
• Triggers: Define when a pipeline should run (manual, scheduled, event-based).

Scenario for our tutorial

To understand the basics, let's create a very simple but common task: copy a file from one location in Azure Blob Storage to another folder in the same or a different Blob Storage account.

Step-by-step instructions: Your first ADF pipeline

Follow these steps to create your first pipeline:

Step 1: Create Azure Data Factory

1. Sign in to the Azure Portal.
2. Click “+Create Resource”.
3. Search for “Data Factory” and select the service.
4. Click “Create.”
5. Fill out the required fields:
   • Subscription: Choose your Azure subscription.
   • Resource group: Select an existing one or create a new one (e.g. rg-adf-tutorial).
   • Region: Select a region near you (e.g. “West Europe”).
   • Name: Provide a unique name for your data factory (e.g. adf-ailio-tutorial).
   • Version: Make sure V2 is selected.
   • (Optional) Configure Git integration, network, and tags as needed (for this tutorial, you can leave the default settings).
6. Click “Review + create” and then click “Create”. Deployment takes a few minutes.

Step 2: Start ADF Studio

1. Once deployment is complete, navigate to your newly created data factory resource.
2. On the overview page, click the Open tile under Open Azure Data Factory Studio. This opens the visual development environment in a new tab.

Step 3: Create Linked Services

We need connections to our source and destination storage.

1. In ADF Studio, click the wrench icon (“Manage”) on the left.

2. Go to “Linked Services” and click “+ New”.

3. Search for and select “Azure Blob Storage”. Click “Next.”

4. Configure the Source Linked Service:

   • Name: Enter a name (e.g. ls_blob_source).
   • Integration Runtime: Leave it as AutoResolveIntegrationRuntime.
   • Authentication Method: Select an appropriate method (e.g. “Account Key”, “System Assigned Managed Identity” if ADF has access to the storage).
   • Azure Subscription & Storage Account Name: Select your subscription and the storage account where your source file resides.
   • Click “Test Connection” to make sure everything is working.
   • Click “Create.”

5. Repeat steps 3 and 4 to create a Target Linked Service (e.g. ls_blob_sink) pointing to the target storage account (can be the same account).

Step 4: Create datasets

Now we define which data (folders/files) we want to access.

1. In ADF Studio, click the pencil icon (“Create”) on the left.

2. Hover over “Datasets” and click on the three dots (…), then click “+ New Dataset”.

3. Select Azure Blob Storage as the data storage and click Next.

4. Choose the format of your file (e.g. “Binary” for a 1:1 copy or “DelimitedText” for CSV). For this example we will use “Binary”. Click “Next.”

5. Configure the Source Dataset:

   • Name: Enter a name (e.g. ds_blob_source_file).
   • Linked Service: Select the ls_blob_source created previously.
   • File Path: Navigate to the container and, if applicable, the folder/file you want to copy. Leave the filename blank if you want to copy an entire folder or specify the specific filename.
   • Click OK.

6. Repeat steps 2-5 to create a target dataset (e.g. ds_blob_sink_folder). Here select the ls_blob_sink as the linked service and specify the target container and, if necessary, target folder. Leave the file name blank as the copy activity will take it from the source.

Step 5: Create Pipeline

1. Hover over “Pipelines” and click on the three dots (…), then click “+ New Pipeline”.
2. Give your pipeline a name in the properties panel on the right (e.g. pl_copy_blob_to_blob).
3. In the Activities pane, expand the Move and Transform section.
4. Drag and drop the Copy Data activity onto the empty pipeline canvas.

Step 6: Configure Copy Data Activity

1. Click the Copy Data activity on the canvas.
2. Go to the “Source” tab at the bottom.
3. Select your source dataset (ds_blob_source_file) from the drop-down list.
4. Go to the “Sink” tab.
5. Select your target dataset (ds_blob_sink_folder).
6. (Optional) Explore the “Mapping” (for mapping columns, relevant for structured data) and “Settings” (for timeouts, retries, etc.) tabs. For our simple example, the default settings are sufficient.

Step 7: Debug/Run Pipeline

1. Click Debug above the pipeline canvas. This runs the pipeline immediately without publishing it (great for testing).
2. Go to the “Output” tab at the bottom. Here you can see the progress and the result of the debug run (status: “In progress”, “Successful”, “Error”).
3. (Optional) To permanently save your pipeline, click Publish All at the top. For example, to run them daily, you could add a “trigger” (“+New/Edit” for triggers).

Step 8: Check result

Navigate to your target blob storage container in the Azure portal or using the Azure Storage Explorer. You should now find the copied file there!

Summary & Next Steps

Congratulations! You have just created and successfully run your first Azure Data Factory Pipeline. You learned how to:

• Create a data factory instance.
• Establish connections to data storage (linked services).
• Define data structures (datasets).
• Create and configure a pipeline with a copy activity.
• Testing (debugging) your pipeline.

Of course, this is just the beginning. Azure Data Factory offers a huge range of connectors and activities, including complex data transformations with mapping data flows, code execution (Azure Functions, Databricks Notebooks), control flow logic, and much more.

Want to dive deeper or need assistance with more complex data integration scenarios in Azure?

Ailio is your experienced partner for Azure data engineering and data science. Contact us to learn how we can optimize your data integration!