Log Shipping

Pyramid logs events in the Pyramid database (as configured in Log Settings) and can also be configured to ship those logs to a third-party log management service, such as Splunk or Datadog. The log management service can be used as simple logging backup or as a full operations tool that allows you to index, search, and analyze logs outside of Pyramid. What is available to you depends on the functionality of your integrated logging service.

The settings in the Log shipping page define details of the target log service and indicate whether to ship system, transaction, or entry logs as needed. Once logs are shipped, you will manage them in your third-party service.

Note: The logs that are collected by Pyramid are also sent to the target log management service. This means that you need to configure both your Log Settings and your Log Shipping details to achieve the desired effect.

Log shipping

Setting up log shipping

Before you begin

Install and configure Splunk or Datadog before you begin this process.

Access the log shipping page

From the Admin console:

  1. In the left-hand menu, click Logs > Log Settings.

    2. The Log Settings page opens. Always check that you are logging those events that you want to ship to the target log management service on this page!

  2. Click the Log Shipping tab at the top of the page.

The Log Shipping page opens (see preceding image). You will set up the details of the target log management service and configure which logs should be sent to that service on this page.

Log shipping settings

Log settings

  • Log shipping enabled: Select this checkbox to enable log shipping.
  • Ship system logs: Send system logs to the log service. These are the system-level messages that are already being logged.
  • Ship transaction logs: Send transaction logs to the log service. Transaction logging tracks query events associated with every report run on the platform.
  • Ship entry logs: Send entry logs to the log service. Entry logging tracks user entry into the application, failed login attempts, and, where the formal logout functions are used in the client, log outs.

Note: The files that are sent to the log management service are the same ones that are logged according to your Log Settings. This means two things: 1) You need to collect system, transaction, and entry logs for them to be sent to the log service. 2) You change the log level of your logged events by changing the log level in your Log Settings.

Log service details

  • Log service: Select the name of the third-party log management service from the drop-down. This can be one of Splunk or Datadog.
Splunk

If your third-party log management service is Splunk, you need to provide the following details:

  • Splunk address: The full secure URL for your Splunk service. Must include the https:// schema prefix, the 8088 port, and the server path. Note: You can click Append server path to append the typical Splunk server path to your URL.
  • Splunk HTTP Event collector token: Paste in this token to enable communication with Splunk.
  • Channel ID: Specify a custom channel ID or click Regenerate to generate one automatically.
  • Validate SSL on connection: When communicating with the log management service, if the SSL isn't valid, Pyramid won't be able to connect or to run a test.
Datadog

If your third-party log management service is Datadog, you need to provide the following details:

  • Datadog domain: This is the domain that is selected for Datadog (where to house your data).
  • API Key: The Datadog API key. This should be available from the Admin panel in Datadog.
  • Source: The name of the source in Datadog. This is an optional field and will default to the name of the Pyramid instance if left blank.

Test and Save

  • Click Test at the top-right of the page to check that your connection details are correct. Note: Clicking Test writes some test logs.
  • Once your test is successful, click Apply to save your settings. If the Log shipping enabled checkbox is selected, the logs will be shipped to the target logging service on being written to the Pyramid database.

Note: If there is a failure to connect to the log management service for any reason, an error will be reported.