You can use Python and R scripts as data sources, by downloading or writing a script and converting it into a table. To connect to a scripting source, add the R or Python node from the Sources menu (not from the Scripting menu) and construct or configure the script from its Properties panel.
Scripting can also be used for data manipulation during the ETL. This functionality is configured by added the required scripting node from the Scripting menu.
- Click here to learn more about Scripting for data manipulation purposes.
Supported Scripting Languages
The following scripting languages are supported as data sources:
Writing a New Script
There are a few ways in which you can provide Pyramid with your script:
- Generate your Script: Produce an AI-generated script based on a given prompt.
- Pyramid Marketplace: Download a script from the Pyramid Marketplace. Pyramid will then import the dataset and convert it to a table.
- Pick a Script: Select a script that's been created in Pyramid and saved to the Pyramid content manager.
- Write a Script: Type or paste a script directly into the Script panel.
Generating a Script
You can generate a script by clicking the Gen AI icon from the Properties panel and, in the Generate Your Script dialog, typing a “prompt” (description of your query) in natural language. Once you are ready, click Run. The AI uses its underlying LLM to generate a script based on your prompt. For more information, see Generate Your Script.
Explain Script
Use the Explain Script function to produce an AI-generated explanation of what the script does (note this is available even if the script was not itself AI-generated). Note: Each time you click the Explain Script button, a new explanation is generated. For more information, see Explain Script.
Scripting Environments
Pyramid enables Administrators to create multiple scripting environments, so as to allow for multiple language and package versions. This means that you'll need to choose which scripting environment to connect to.
When choosing the environment, you'll be able to view a list of each package (and its version) currently installed on the selected environment.
Output and Data Frame Support
After providing the script, you'll need to define the output, including the data frame, column name, and data type for each column. Pyramid automatically enters the outputs for scripts downloaded from the Marketplace. For other scripts, you can configure the output manually column by column, or you can use the Auto Detect data frame support function. Auto Detect lets you provide the data frame once, and then Pyramid detects the data frame and columns in the script and adds them to the output.
- Click here to learn more about data frame support.
