Monitor system health
- Gisela Roberts
- Dev Ops
- Andy Roberts (Deactivated)
- Ben Stanley
To monitor system health, first view the available information about the tracking, position and rotation of the tracked objects. For a more detailed examination of any possible issues, you can then use Evoke's main health metrics.
You can obtain additional information about camera health from accelerometer feedback (see Observe accelerometer feedback) and from examining the camera temperature monitoring (see Monitor camera temperature).
When you have determined whether any of the cameras need further action to correct any issues, see Fix issues with camera calibration.
Plot graphs of object count and object pose
The main indicator of the camera calibration being healthy is that objects are successfully tracked and their poses (ie, their position and rotation) are stable and noise-free in all parts of the working motion capture volume.
You can display this information by plotting graphs of object count and object position/rotation using the Graph Plots view (see About the Graph Plots view and Graph your data).
However, it may not always be obvious that the tracking is failing for a few frames or that the pose of the object is noisy in particular parts of the volume, particularly when dealing with large numbers of objects. In these cases, examining the primary camera calibration health metrics may indicate potential issues. For details, see Examine the main health metrics.
Examine the main health metrics
In cases where the Graph Plots view does not reveal the source of any issues (see Plot graphs of object count and object position/rotation), the following primary camera calibration health metrics can help to reveal potential problems.
You can view these scores for all cameras and individual cameras in the System Health Report panel (see Understand the System Health Report), and for individual cameras in the Cameras view (see About the Cameras view).
Centroid connection score
The centroid connection score is the percentage of centroids that are connected to labeled object points, calculated over a fixed time window. To be included in the centroid connection score, centroids must be connected to object points.
Recommendation for strobed systems
If your system includes cameras with built-in strobes (such as Vero, Vantage, or Valkyrie cameras), we recommend that you turn on online masking to filter static centroids that aren't connected to object points (eg, from unused passive markers at the edge of the volume, or from unmasked reflections on a metallic camera truss) from inclusion in this score.
To turn on online masking of these unconnected centroids, in the Processing panel, select Advanced Parameters 
, go to the Camera Healing section and select Generate Online Masks.
Image error score
The image error score is the average error for centroids that are connected to labeled object points. The error here is the distance (in pixels) between a centroid and the projection of the associated labeled object point into the associated camera view.
The following examples illustrate common calibration scenarios, each showing a zoomed section of a Cameras view:
|
Example |
Calibration issue |
|---|---|
|
|
Good/perfect calibration The centroids are displayed as circles with internal crosses to help show the exact centers. All centroids are connected to labeled object points (indicated by the green lines) and so the Centroid Connection score is high (100%). The centroid centers line up with the projections of the object points in this camera view, and the Image Error score is very low. |
|
|
Noisy calibration The estimated calibration for this camera no longer perfectly matches the true calibration. The calibration is close enough that all the centroids can still be connected to the corresponding labeled object point, but the projections of those points are further away from the equivalent centroids. This results in a high (good) Centroid Connection score, but the Image Error score is higher (worse) than normal, indicating a problem. This situation might result in some extra noise/jitter in the object's path, particularly as the set of cameras that can see that object change, but in general the object would still be expected to track. |
|
|
Bad calibration (drift) The true calibration of the camera has drifted since the initial calibration. Even though the difference is relatively small, it is large enough that that none of the centroids are connected to object point. Unconnected centroids are indicated by being drawn in red. The Centroid Connection score is very low (0% connected). As there are no connected centroids, it is not possible to compute the Image Error score. Depending on the number of cameras similarly affected, the tracking may start to show more severe noise/jitter and the object might start to occasionally fail to track in some parts of the volume. |
|
|
Bad calibration (bump/knock) The true calibration of the camera significantly differs from the software's stored calibration. This may have been caused by the camera being knocked or deliberated moved since it was last calibrated. The Centroid Connection score is very low (as in the Bad Calibration (drift) example above) and again the Image Error score is not computed. The cameras affected to this extent do not contribute at all to the tracking of objects. This might result in the object failing to track in parts of the volume (depending on the extent of coverage redundancy, primarily the overlap between cameras view frustums). |
For information about the meaning of the colors of the centroids in the Cameras view, see About the Cameras view and to find out how to change your view of the centroids, see Set appropriate View Filters.
Observe accelerometer feedback
When a camera's accelerometer measurement exceeds its bump threshold, bump notifications are shown in the System Health Report.
Where it is possible to use automated healing, these warnings can largely be ignored; but in certain situations it can be useful to use accelerometer bump notifications to identify problematic cameras and then use the manual Recover Camera Pose operation (see Use Recover Camera Pose) to correct those cameras.
When the bump notifications are required for the manual process of identifying the problematic camera(s) and then using Recover Camera Pose, it may be useful to enable their display in the System panel and Workspace camera views.
To display bump notifications in the System panel and Cameras view:
- From the Settings menu, select Preferences.
- Go to the System Health section and select Show Bump Notifications Per Camera.
Bump notifications are now displayed for individual cameras in the Vicon Cameras list in the System panel.
On Vicon cameras with accelerometers, the status indicators flash blue when the camera is bumped and (on Valkyrie and Vantage cameras only) the display changes to indicate a bumped camera.
In addition to the bump notification warning, the System Health Report also provides a Camera Accelerometer section, which displays a graph showing the maximum accelerometer signal for any camera. This is useful for showing when the cameras accelerometers are detecting motion that may not trigger the bump notification warning but may still be of interest or concern, eg, a door slamming may cause the cameras to momentarily shift but will likely return to their previous position, and the tracking may be affected for a very limited period.
Monitor camera temperature
Several common issues with camera calibration arise from changes in temperature. Ambient temperature changes may cause thermal expansion/contraction of the camera rig causing camera positions to change from their calibrated values. Internal changes of temperature can lead to internal shifts of lens elements and the sensor that may invalidate calibrated parameters.
For this reason, Vicon cameras, have onboard temperature sensors that can be used to help monitor temperatures. The System Health Report contains a Camera Temperatures section (see Camera Temperatures), which collates all the temperature sensor information for the cameras that can be found in a camera's properties in the System panel (see Vicon Cameras settings).
Generally if there are no observable problems in the tracking of objects and the camera calibration health metrics such as Centroid Connectivity and Image Error do not indicate a problem then changes in temperature can be assumed to be harmless. The temperature warnings are most useful in helping to diagnose issues.
