The Scripting Environments manager allows admins to create and govern multiple virtual environments for Python and managed environments for R. The environments are used to stage and run Python and R scripting in Model, Discover and Formulate. Environments can be added, removed, and edited.
Why Use Environments?
Versioning
Both Python and R are scripting languages. Neither has inherent versioning switches and both use fixed version third-party packages. As such, it is extremely complicated to create and run different Python or R programs from the same setup, which each program may use a different version of the language engines AND different versions of the underlying third-party packages.
Pyramid solves this issue by creating and maintaining multiple environments:
- Python - Multiple virtual environments can be created and deployed to one or more AI servers. Different versions of Python CAN coexist on the same host, so it is possible to deploy multiple coexisting versions.
- R - A managed environment can be created and deployed to each AI server (or duplicated across multiple servers). Different versions of R CANNOT coexist on the same host, so while there can be multiple environment definitions of R in Pyramid, each one can only be deployed to any given host at any one time.
Scale Out and Load Balancing
Each environment can be hosted on multiple AI servers so the mechanism also effectively provides an load balancing and resource allocation framework for Python and R processing as well.
Environments Manager
The Environments manager lists the current environments:
- Type: Python or R.
- Environment: The name of the environment.
- Status: The status of the environment; green for installed, yellow for installing / downloading packages, and red for not usable.
- Version: The Python or R version.
- Servers: The list of AI servers on which the environment is hosted.
For each environment, administrators can access the following Actions:
- Edit: Opens and edits the given environment in the bottom panel.
- Hosted Servers: Set and edit which AI servers to host the environment on.
- Reinstall: This option is enabled if the environment is unstable; for example, if a package failed to download or a server crashed. In this scenario, the environment may be reinstalled.
- Delete: Deletes the selected environment.
- Duplicate: Create a duplicate of the selected environment.
The "Pyramid Python Environment" and "Pyramid R Environment" are created during installation and may not be edited or deleted. However, they can be moved to operate on different AI servers.
Add a Virtual Environment
To add an environment, click Add Environment or Add Existing Environment. The Environment Settings panel is opened by adding an environment or editing an existing environment
Environment Settings
- Secure: if the environment is going to be enabled for certain tenants only, check the box and choose which tenants will have access to the environment.
- Name: provide a name for the environment.
- Type: select either R or Python.
- Version: select the appropriate version for the given type.
- Description: add a description if required.
- Server Coverage :This setting is not visible unless the automatic "Percentage" scaling model is chosen. If provided you must set the percent "coverage" that this environment will have on all AI servers in the cluster. Setting to 100%, means the environment is deployed to all machines. 50% means it's deployed to half the servers etc.
Add Packages
The options for adding packages are as follows:
- If you are installing Python, you can install from an Archive File, a Custom Repository, or from the Default Repository. The Package Managers for this type of installation are Conda or Pip, although Archive File installation is Pip only.
- If you are installing R, you can install using a Custom Repository or from the Default Repository. The package manager type is ignored for R installations.
Install a Python or R package from the Default Repository
Important: The repository that is used when you install a Python package using Conda is defined in your DSML Settings as the value of Default Conda Repository. This may be using the Conda-Forge repository or switched to using Anaconda.
Select the following options:
- Package Manager: For Python installations, select either Conda or Pip.
- Install from: Select the Default Repository option.
- Name: The name of the package to install.
- Version: Optionally, specify the package version. If this field is left blank, the latest version of the package is installed.
Install a Python package from an Archive File
Installation from an Archive file is available when installing a package for Python.
Select the following options:
- Package Manager: Select Pip.
- Install from: Select the Archive File option. This allows upload of packages / custom packages directly to Pyramid.
- Archive file: Navigate to and select the archive file.
Note: It is important to note that all dependencies must be known and must also be uploaded.
Install a Python or R package from a Custom Repository
Installation from a Custom repository is available for both Python and R.
Note: When you install Python packages using Pip from a Custom Repository, and the repository uses SSL and has a self-signed certificate, you need to upload that certificate to the Certificate Manager to be able to perform your installation. Important: Once you have uploaded your certificate, you must restart the services as prompted by the Certificate Manager before you can install your package.
Select the following options:
- Package Manager: For Python installations, select either Conda or Pip.
- Install from: Select the Custom Repository option. This allows any package repository to be used, such as a custom on-site repository.
- Repository: Provide the URL for the package. Note: When the package being installed is a python package and the Package Manager is of type "Pip," this must refer to a "Simple" Pip repository's URL.
- Name: Specify the name of the package to install.
- Version: Optionally, specify the package version. If this field is left blank, the latest version of the package is installed.
Package Listing
The Environment Settings panel lists all the packages downloaded to the given environment, along with the package version, package manager, and any additional information.
From here you can update a package version or remove or download a package.
- Actions: Edit the name, version, or package manager, or delete a package.
- Packages: Package name.
- Version: Package version.
- Package Manager: Whether the package was downloaded from Pip or Conda.
- User Defined: If yes, the package was imported by a user. If no, then it is a dependency package or a system package. For instance, if the user imported numpy, the numpy package is user defined; the setup tools package is a dependency package; DateTime is a system package.
- Info: indicates any additional information about the package.
Hosted Servers
This tab is visible if the manual scaling model is chosen. Then, admins must manually allocate each environment to each AI server in the cluster. Scripting environments may be installed on multiple servers.
By default, the environment will be hosted on the tenant's default AI server.
Finish
Once you've selected the required packages, click Add to complete the installation