Understand System Health Report

System Health Report is a default pane in Tracker. It consolidates existing key metrics into a single, easy-to-understand format to help you monitor and evaluate overall system performance. It also provides visibility into cameras currently undergoing healing processes.

The default view of the System Health Report features multiple section headers which provide an insight into the current system performance. Each section is expandable so that you can access more detailed information about the metrics presented in its header. The metrics in each section are presented either as a discrete value at the current frame or as a thumbnail displaying a trend of recent data in a specified time window.

All metrics presented in the System Health Report can be monitored in the app and accessed remotely via the API.

The following descriptions outline each section, and underlying metrics, in the System Health Report. This can help you decide which metrics to display while tracking.

Performance section

In the System Health Report, the Performance section helps you to assess system performance using data latency and drop frames. These metrics provide an idea of the system's current responsiveness and reliability.

• Latency This refers to the time elapsed (in ms) measured from the mid-point of the camera shutter interval to the moment data is ready to be sent to a client.
• Drop Frames This is the number of unprocessed frames expressed as a percentage of the total frames. Drop frames occur when the processing of the frame is longer than the frame interval itself. This may happen, for example, when tracking high object counts or when the frame rate is sufficiently high.

The section header provides the current system latency and whether the system is currently processing with or without Drop Frames.

When this section is expanded, it displays:

• A thumbnail graph displaying the latency of the datastream output for the most recent 60 seconds. It is split into the total latency (white) and the delivery latency (red).
• A table that displays the current system latency as described above and the Drop Frames, expressed as a percentage, over the last 60 seconds.

To display the latency presented in the Performance section in Graph Plots, select the Datastream Output Latency channel type.

If you want to tweak the performance to affect latency or drop frames, consider changing the settings in the Processing panel (Configure system processing parameters).

Calibration Details section

In the System Health Report, the Calibration Details section provides details about the state of the system's calibration. The section header indicates whether all enabled cameras are calibrated.

When this section is expanded, it displays:

• The time elapsed since last calibration
• The number of enabled Vicon cameras that are uncalibrated.

If uncalibrated cameras are displayed and you want them to contribute to tracking, be sure to perform a calibration.

Centroid Connectivity section

In the System Health Report, the Centroid Connectivity section helps you to evaluate tracking fidelity using a metric called Centroid Connectivity (also referred to as 'Centroid connectivity score' or just 'score'). The score is the total fraction of moving 2D centroids (from each individual camera) that can be labeled as object marker centroids as a percentage of total moving centroids (see also Centroid connectivity score). A low score indicates poor object tracking.

The section header displays the number of cameras whose centroid connectivity is below 50%. The icon also displays a gradient from red (poor score) to green (perfect score).

When this section is expanded, it displays:

• A thumbnail graph that shows the Centroid Connectivity for the most recent 60 seconds to help assess any potential volatility in tracking.
• A table showing average centroid connection score
• A list of all cameras that are currently healing in an attempt to improve its score. This is only displayed if Auto Bias Handling or Auto Recover Camera Pose is selected in the Processing panel or in the System Health Report. The headings in this list include the camera number, the current score of that camera and the total moving centroids in that 2D view. Cameras continue to undergo healing until its score reaches 100% or the parameters are disabled.

If your centroid connectivity is low, your capture environment or the geometry of your tracked object may have changed. To troubleshoot, consider the following solutions:

• Ensure there aren't any moving reflections in your volume
• In the Processing panel or in System Health Report, turn on Enable Bias Handling.
• In the Processing panel or in System Health Report, turn on Enable Auto Recover Camera Pose.
• Re-calibrate the system (particularly if there is a large deviation in camera temperatures).
• Re-evaluate the tracked object.

You can also display Centroid Connectivity in each camera's individual view. For more information, see About the Cameras view.

Image Error section

In the System Health Report, the Image Error section helps you to assess the accuracy and precision of the current state of tracking. Image Error is a metric that calculates the difference between 2D centroids and their expected location in the image (based on 3D re-projection) expressed in pixels (see also Image error score). This metric helps you to evaluate the quality of the connected centroids and is the same metric used to assess the quality of the dynamic calibration. As such, if the image error is increasing, a poor calibration or a change in object geometry may be indicated.

The header displays the number of cameras with an Image Error above the threshold (1.0 pixels).

When this section is expanded, it displays:

• A thumbnail graph with the average Image Error across all cameras for the most recent 60 seconds
• A table with the current Image Error
• A list populated with the same cameras as the Centroid Connectivity section. This list shows the corresponding image errors for the cameras and the number of labeled centroids. As a camera improves its score, the Image Error will normally also increase. In general, a large Image Error may not be desirable if high accuracy and precision is required.

Camera Temperatures section

In the System Health Report, the Camera Temperatures section helps to monitor the body temperatures of each camera connected in the system. This section provides details about potential changes in your capture environment since the system was calibrated. If the system is displaying poor Centroid Connectivity and/or high Image Errors and there is a large deviation in temperature between the current state and when the system was last calibrated, this can indicate that you must perform a new calibration.

The section header can display two different metrics:

• If the system is calibrated, it shows the average temperature change of all cameras since calibration.
• If the system is uncalibrated, it shows the current average temperature of all cameras.

When this section is expanded, it displays:

• A thumbnail graph that displays two temperatures for the most recent 30 minutes: the average camera temperature at calibration (white) and the current camera temperature (red). All values are expressed in degrees Celsius.
• A table that shows the minimum, maximum and average value for three temperatures:
  • Change during calibration: This is the change in camera body temperature from the start of calibration. You may not see a difference in this row unless you have a large volume or high camera counts to calibrate or an unstable environment.
  • Change since calibration: This is the change in camera body temperature since the system was last calibrated.
  • Current temperature: This is the current camera body temperature of all cameras.
• A list showing a summary of temperature changes for each camera both during and since calibration.

Camera Accelerometers section

In the System Health Report, the Camera Accelerometers section helps to monitor the accelerometer readings for all cameras. The accelerometers are used to indicate that the camera body has accelerated beyond its threshold (see Bump Sensitivity in Camera properties). It is a crude way to indicate that a camera may have shifted position and thus reduced tracking fidelity.

The section header displays the number of active bump notifications identified in the system.

When this section is expanded, it displays:

• A thumbnail that displays the maximum acceleration measured on any camera within the last 10 minutes.
• A table that shows the number of cameras with an active bump notification (see Observe accelerometer feedback).
  
  

To change the sensitivity of the camera accelerometer, in the System panel, select the camera and in the advanced properties, go to the Accelerometer section and change the Bump Sensitivity.

Auto Recover Camera Pose section

In the System Health Report, the Auto Recover Camera Pose section is used to track any cameras that have been selected for automatic camera pose recovery according to some of the other parameters in the System Health Report. For more information, see Auto Recover Camera Pose.

The header provides the option to enable/disable the Enable Auto Recover Camera Pose parameter, which is also found in the advanced parameters of the Processing panel (see Configure system processing parameters).

Auto Bias Handling section

In the System Health Report, the Auto Bias Handling section is used to track any cameras that have been selected for automatic system healing according to some of the other parameters in the System Health Report. For more information, see Auto Bias Handling.

The header also features the ability to enable/disable the Enable Auto Bias Handling parameter located in the advanced parameters of the Processing pane.

When expanded, it displays the average Image Error Threshold across all cameras.