The Data Flow is where you construct your data flow diagram, representing the flow of data from the datasource to the target. The first step in the data flow is to connect to one or more datasources; you can then copy the required tables into the data flow, or enable direct querying for the source. Whether you choose to ingest data (copy tables from a source and load them into a target) or directly query the data source, the Data Flow contains the instructions of what to do with the tables selected from the data source.
If you opt for data ingestion, the Data Flow may also contain instructions for data preparation and manipulation.
- Click here to learn about the Data Flow.
The Master Flow must contain at least one data flow, and can contain multiple data flows.
Add a Data Flow
When switching from the Data Flow panel to the Master Flow Panel, the current data flow is automatically added to the master flow, even if you haven't yet connected to a source.
You can add more data flows by dragging the Data Flow node onto the canvas.
Multiple Data Flows
You can reference multiple data flows in a single master flow. To open the data flow panel for a given data flow node, click the node before going to the Data Flow panel. The selected data flow will be shown.
Data Flow Properties
Click on a Data Flow node to show its properties in the Properties panel.
Display Name
By default, the node's display name is 'Data Flow 1'. Any subsequent data flow nodes are named according to this naming convention, with the appropriate numeric suffix, e.g. 'Data Flow 2', 'Data Flow 3', etc.
You can change the data flow node name from directly from this field.
Description
You have the option to add a description to each data flow node; this can be a useful way of documenting the node for yourself and other users.
