Parquet File Source

Pyramid allows you to upload or connect to Parquet files and convert the data into a table. Each given Parquet file is converted into a single table. The Parquet file source supports local file upload, or pointer to a shared file on a network drive, or URL address.

Provide a Parquet File Source

Start by adding the Parquet source node and go to its Properties panel. Connect the required file by uploading it, or pasting a pointer to a shared file, or a URL address.

Uploaded File

Upload the file by dragging it from its folder location onto the upload widget. Alternatively, click Upload File to open your file explorer and find and select the file from its folder location.

Shared File

Enter the file path of the shared file (including the file name and extension) in the File Path field (green highlight below). Enable 'Expression' (yellow highlight) to provide the file path in the context a dynamic PQL expression, created in the PQL Editor.

  • Click here to learn more about connecting to a shared file.

URL

The data will be downloaded from the URL when the model is processed.

To connect to a Parquet file via its URL, paste the URL in the Set URL field. Enable 'Expression' to provide the URL in the context a dynamic PQL expression, created in the PQL Editor.

Select the appropriate authentication type:

  • None: select None if no authorization is required
  • Basic Authentication: if basic authentication is required
  • Custom Header: if custom header authorization is required

Configure a Parquet File Source

Start by adding the Parquet source node and go to its Properties panel. Connect the required file by uploading it, or pasting a pointer to a shared file, or a URL address. Once you've uploaded or connected to the file, you'll need to configure the properties to determine the structure of the table.

  • Encrypted: If the Parquet file has been encrypted, provide the key to enable Pyramid to decrypt the file.
  • Change Source: upload or connect to a different Parquet file.
  • Update Table: update the resulting table with any changes made to the JSON properties. This option appears only after the table has been added to the data flow.

Table Selection

The Parquet file will be converted into a single table; select the table and click the 'Add Table' button.

When the new table is previewed, we see that each Parquet field name has been converted into a column name, with each column containing all the values for that field name: