Azure Data Factory allows users to reference set values in the configuration of the pipeline, which can be referenced within multiple “activities” along the pipeline. The two main methods are called parameters and variables, which can both be created in the configuration pane found by clicking on the background canvas of any pipeline. While these can look similar to a new user, the use cases for each are very different. Here are the differences, and some common methods of assigning variables in ADF.
In short, variables can be changed during a pipeline, whilst parameters are static while the pipeline runs.
Parameters
Parameters are constants which hold either a single value or an array of values. They are typically used to apply common configuration settings across the pipeline, which are directly related to the pipeline’s function. For example, a parameter may be used to make sure a file prefix is consistently applied to all data inputs. Parameters can be used for most standard data types as well as SecureStrings, which mask the value of the parameter for use with inputs like passwords.
Parameters can also be updated when one pipeline (let’s call it Pipeline A) is executed from another pipeline (Pipeline B). This value can be allocated within the triggering pipeline (B), allowing for a single pipeline (A) to be repurposed for any number of configurations assigned by the parameter. When using the “ Execute Pipeline” activity to trigger one pipeline from another, the pipeline parameters are shown in the settings pane, which can then be altered or left to the default value.
Variables
Variables are typically used to assign values within a pipeline, and can be changed during the running of a pipeline. Variables only hold one value (or one array) at a time, but can also be assigned more than one value during the running of a pipeline, allowing them to be reused within iterative processes. When configured as arrays, values can be added to an array during the running of a pipeline. This flexibility allows for the pipeline to change during each run, making pipelines dynamically change with some characteristic of the input data or the pipeline itself.
Assigning Variables
Variables are typically altered using two pipeline activities; “Set Variable” or “Append Variable”.
“Set Variable” is more common, and assigns a new value to the variable selected. This overwrites the previous value, and since it can only show one value, only shows the most recent set value.
“Append Variable” is more niche, and allows for an additional value to be added to a variable set to “array” type. The variable can only hold one array, but arrays can contain multiple values, allowing for pipelines to build up a collection of values which can be referenced in other processes.
Either of the above activities can be used multiple times in a workflow, or as part of iterative processes, which can lead to a single variable being assigned hundreds of values in a single pipeline. It is worth noting that arrays have a maximum number of values set at 100,000, which may limit the usefulness of arrays for fields where many unique values are expected, particularly from fact tables.