About Vicon Shogun 1.6

Shogun 1.6 is the sixth point release of Vicon’s entertainment market software.

Shogun Live now provides virtual production tools to support low latency, high accuracy camera tracking mixed with full-body tracking with Unreal integration. It also includes new supported SDI cameras, Courtyard timecode generators, and Bluefish capture card support. In Shogun Post, Auto-Skeleton provides greater control over the solving subject as a post-processed alternative to the Live calibrated solve.

For a more detailed description, see:

• New features in Vicon Shogun 1.6
• What's new in Shogun 1.6 on YouTube.

This release also benefits from ongoing maintenance, with a number of issues having been addressed (see Addressed issues in Vicon Shogun 1.6).

For information on requirements for installing and running Shogun, see PC requirements in Installing and licensing Vicon Shogun.

For detailed information on PC requirements, visit the Vicon website FAQs and select Operating systems and PC or contact Vicon Support.

New features in Vicon Shogun 1.6

The following sections describe the new features in this release of Shogun.

• Shogun Live 1.6 new features
• Shogun Post 1.6 new features

Shogun Live 1.6 new features

These are the main new features in this release of Shogun Live:

• Features added to support virtual production
• Full Bluefish444 capture card support
• Courtyard timecode generator support
• Viper and ViperX support
• Monitor datastream connections in the Connections panel
• Check for a successful wand wave

Features added to support virtual production

These features are provided to aid your work in virtual production.

Low latency object tracker

Shogun Live 1.6 offers you the option of selecting Vicon's low latency, robust object tracker, which is already available in Vicon Tracker and Vicon Evoke, to track rigid objects such as camera-tracking crowns or props.

Traditional object tracking requires at least three cameras with an unoccluded view of a marker to be able to reconstruct it in 3D. Instead of this approach, the object tracker starts tracking by using the 6 DoF pose of the rigid object itself (ie, the position of the markers in relation to each other) to generate reconstructions and labeling. The generated reconstructions are used by the object tracker to boot a solution but are not needed on every frame, allowing the object tracker to generate results even in situations where traditional object tracking drops frames.

By tracking using the relationships between rigid points from 2D data, Shogun produces tracking that is robust to occlusion and only requires the cameras to be able to see three markers from an object. Latency is also minimized as it is not necessary to wait for reconstruction, labeling and solving to complete before an object pose is obtained.

To use the object tracker:

1. Create the object in the usual way:
   1. In the 3D Scene, select the relevant markers.
   2. In the Tracking panel, with the Setup tab selected, go to the Prop field, enter a name for the object and click Create.
2. On the Properties tab, ensure Track with ObjectTracker is selected.

   

For more information, see Understand object tracking in Shogun Live in Getting more from Vicon Shogun.

Force Lowest Latency option

To enable you to achieve the lowest latency possible for object tracking, you now have the option of selecting or clearing the Force Lowest Latency setting in the System panel. Selecting this option provides the lowest possible latency at the cost of reducing grayscale and centroid throughput. However, it is normally only useful if you are using low-latency object tracking (see Low latency object tracker).

Note that this feature is not available for Vicon cameras that do not have a DSP, such as Vicon Viper and Vero.

For more information, see the Use the Force Lowest Latency option section in Understand object tracking in Shogun Live in Getting more from Vicon Shogun. .

Ability to use object presets

Object presets enable you to specify the object tracking and smoothing (filtering) properties for selected objects. (Note that setting these properties overrides the overall object tracking properties that you can set in the Processing panel.)

By experimenting with different values for the filtering properties and applying them to an object using a saved object preset, you can evaluate filter performance to ensure the smoothest possible tracking for selected objects.

To create and apply object presets:

1. In the Tracking panel, ensure the object whose tracking and/or filtering properties that you want to refine is selected.
2. On the Properties tab below, ensure the Advanced properties are displayed and in the General section, on the right of the Object Preset field, click Manage.
3. In the Object Presets dialog box, enter a name for the new preset, click Add and ensure the new preset is selected.

   

4. Specify the required settings and close the dialog box. Object presets are saved in the Subjects.mcp file (found in C:\Users\Public\Documents\Vicon\ShogunLive1.6\LastRun\UserName), or in any exported tracking configuration.
5. In the Tracking panel, ensure that the object to which you want to apply the preset is still selected and that Advanced properties are still displayed.
6. From the Object Preset list, select the preset that you created. The new tracking and/or filtering properties are applied to the object.

Calibrate selected cameras

WIth Shogun 1.6, you can mask and calibrate or recalibrate multiple selected Vicon optical cameras or Vicon video cameras, as well as SDI cameras, without having to completely re-calibrate the whole system.

Any cameras that are not selected are locked, so that any existing calibration is unaffected, and the global co-ordinate system is maintained.

To add new cameras to a calibration:

1. Connect the new cameras into the Vicon Shogun system that contains the calibrated cameras.
2. In the System panel, select the new cameras that are not yet calibrated (indicated by a cyan icon).

   

3. Instead of masking all the cameras, on the Camera Calibration tab, ensure the Advanced options are displayed and in the Auto Masking section, change Cameras To Mask to Selected Cameras and then click Start (Selected).

   

4. When the cameras are masked, click Stop.
5. In the same way, with the new cameras still selected, in the Calibration Wave section, click Selected and, with Auto Stop selected, wave the wand for just the selected cameras and wait for the calibration to finish.

You don’t need to re-set the origin as the world co-ordinate system remains valid.

SDI lens calibration example

You can use the new functionality to calibrate an additional SDI camera.

With Shogun Live, you can obtain the camera extrinsics (the camera position relative to the world) and the intrinsics (information about the lens, such as its focal length and radial lens distortion). As this enables you to accurately overlay 3D data onto the 2D data, it provides a quick way to assess the quality of the calibration.

To add an SDI camera to a calibration:

1. Connect the SDI camera into a Blackmagic Decklink card on the PC that is part of a Vicon Shogun system. It is displayed as Decklink in the Video Cameras section of the System panel.

   

2. Click Decklink to display the video stream.

3. As in Calibrate selected cameras, mask the selected additional camera and complete a wand wave for it.

   Tip: When waving the wand for the new video camera, bear in mind that you also need several other optical cameras to be able to gain enough information from the wand wave to be able to position the additional camera within 3D space successfully. You may find that holding the wand at an angle may give a better result.

   When the wand wave is complete, the SDI video camera is calibrated and added into the existing calibration. It is displayed in the 3D Scene view. Because the calibration has provided extrinsics as well as intrinsic data about the camera, you can accurately overlay 3D data on top of the 2D data. This gives a quick way of assessing the quality of the calibration as, using the 3D Opacity control in the View Filters, you can see how closely the 3D image matches the 2D data beneath.

   

   Because this process is so fast and consistent, you can make changes to the lens (eg, use a different focal distance) or even swap it entirely and produce a new calibration very quickly.

Link prop and camera

You can now link calibrated SDI or Vicon Vue video reference cameras to a rigid tracking object. This enables the video camera to be tracked, providing real-time 3D overlay in Shogun Live while the camera is moving and enabling the animation to be captured to the MCP.

To link a calibrated video camera and a rigid tracking object:

1. Create the object (normally a tracking crown or set of markers attached to the camera) in the usual way:
   1. In the 3D Scene, select the relevant markers.
   2. In the Tracking panel, with the Setup tab selected, go to the Prop field, enter a name for the object and click Create.
   3. If you want to use the object tracker (see Low latency object tracker), on the Properties tab, ensure Track with ObjectTracker is selected.
2. At the top of the Tracking panel, ensure the object is selected.
3. In either the Workspace view pane or the System tree, Ctrl-click to select a calibrated video camera.
4. In the Tracking panel, on the Setup tab, click Link Camera.

   

    The button text changes to display identifying information for the linked camera and the bone for the prop moves to the video camera's optical center, aligned with the video camera's coordinate system.You can now move the camera while maintaining the 3D overlay, and you can see the camera moving in the 3D Scene and Cameras views.

To detach a video camera and rigid tracking object:

1. At the top of the Tracking panel, select the prop that you want to unlink from the video camera.
2. On the Setup tab, to the right of the Link Camera button, click Unlink. The prop and camera are unlinked, and no longer move together. The prop origin remains at the video camera optical center, and the video camera remains in its last position.

If you recalibrate the linked video camera, for example, following a zoom lens or tracking crown location change, the linked prop is also automatically updated so that the correct calibrated nodal point is maintained.

After you have captured a scene with linked video cameras, you can load the MCP into Shogun Post. The linked video cameras display the same animation in Post, with a key per frame for translation and rotation.

Support for static meshes

With Shogun Live 1.6, you can import objects (FBX files) that represent set pieces and LED display walls into Shogun Live and manipulate them within your scene. This is useful when setting up your 3D environment, enabling you to:

• Calculate the positions of scenery and LED display walls within the tracked space.
• Position actors within the tracked space.
• Approximately place cameras ahead of a shoot.

To import a static mesh into your scene:

• At the top of the Tracking panel, click the Import Mesh button , locate the required FBX file, and click Open. When meshes are imported, for convenience they are copied to C:\Users\Public\Documents\Vicon\SceneMeshes.

To change the mesh to a previously imported mesh:

• In the Tracking panel, select the object, and on thePropertiestab, click theMesh list and select the required mesh from the list.

You can translate and rotate the object mesh in the same way as you manipulate other props (see Move props in Getting started with Vicon Shogun).

After you have finished capturing your scene that contains a static mesh, you can load the resulting MCP file into Shogun Post. For information, see Ability to load static meshes from Shogun Live.

Ability to export lens maps

You can now export lens maps (ST map as a 32-bit EXR file) for calibrated video cameras as distortion maps. You can then import your calibrated video camera/lens into a game engine such as Unreal, or into your compositing software.

To export an ST map:

• On the File menu, click Export Video Camera STMap.

  

  The file is saved in the format <Camera DeviceID>_<focal length in pixels- integer part>_<focal length in pixels - decimal part>_<date time stamp>.exr.

  STMap contains information about the distortion of the lens and has four channels: R,G,B,A:

  • B and A store the location of the pixels in a pinhole/undistorted image, ranging from 0 to 1 in proportion to the width and height of the image. These two channels are used to distort the CG image to match the SDI camera output.

  • R and G store the location of the pixels in a raw/distorted image (currently unused).

This is an example of an ST map:

For information on how to use an ST map in an AR/Simulcam workflow, see Use an exported ST map in Getting more from Vicon Shogun.

Full Bluefish444 capture card support

Shogun Live 1.6 includes support for Bluefish444 SDI video capture cards.

For information on setting up these cards, see Set up a Bluefish444 card in Getting more from Vicon Shogun.

Courtyard timecode generator support

With Shogun Live 1.6, you can set up the Courtyard CY440 to provide reference and timecode to both an SDI camera and the Vicon system.

For more information, see Set up a Courtyard CY440 timecode generator in Getting more from Vicon Shogun.

Viper and ViperX support

Shogun 1.6 supports Vicon's strobeless cameras, Viper and ViperX, in addition to Vero and Vantage cameras, providing full flexibility for tracking volume configuration for active-only or mixed passive and active tracking.

Monitor datastream connections in the Connections panel

You can now display the current connections for the datastream ports. To do this:

• On the View menu, click the new Connections option to display the Connections panel.

  

Check for a successful wand wave

In Shogun Live 1.6, you are warned if your wand wave has insufficient spread across cameras to give a good calibration.

To view the information, look in the Log. The warning displays both the camera User ID (the number you can provide for the camera in the camera's properties, and URN (the Device ID, found in the camera's Advanced properties).

Shogun Post 1.6 new features

These are the main new features in this release of Shogun Post:

• Auto-Skeleton – a post-processed alternative to Live calibrated solve
• Interactive solving
• Write out a video file of the viewport
• Ability to load static meshes from Shogun Live
• Export retarget constraints to HSL
• New hot keys for retargeting and labeling
• BVH export
• New and changed HSL scripting commands

Auto-Skeleton – a post-processed alternative to Live calibrated solve

Live calibration produces reliable results for accurate labeling and can produce a high quality solve ready for direct real-time previsualization or retargeting. Auto-Skeleton enables you to intervene and have greater control over the solving subject as a post-processed alternative to the Live calibrated solve. You can re-import the solving subject into Shogun Live so that you can use this for real time and also capture of processed data.

To use Auto-Skeleton:

1. In Shogun Post, load a ROM or at least a labeled T- or A-pose.
2. In the Subject Setup panel, click the Solving tab, then Actor Setup and select Use Auto-Skeleton.

   

3. To generate the solve, click AS Calibrate.

For more detailed information on how to use Auto-Skeleton, see Use Auto-Skeleton in Getting More from Vicon Shogun.

Interactive solving

If you want to perform a simple manipulation of a solving bone or a retargeting bone, you can now save time and a lot of manual tweaking by using the new interactive solving feature. This immediately updates the offsets for the constraints that drive the bone that you're adjusting as soon as you release the mouse button.

To manipulate a solving bone:

1. Ensure you have loaded the subject, which must have a solving skeleton.
2. In the Subject Setup panel, click the Solving tab, and on the Modify Setup tab, expand Constraints in Current Subject.

   

3. In the 3D Scene, select the joint you want to rotate.

4. On the right of the the Subject Setup panel toolbar, click the Enable interactive solving of the solving subject button . At the top of the 3D Scene view, the text Interactive Solving is displayed.

5. Using the manipulator, rotate the subject's joint.
6. When you're happy with the new position of the joint, release the mouse button. The offsets for the constraints which drive the bone are updated.

You can follow a similar procedure to manipulate retargeting bones:

To manipulate a retargeting bone:

1. Ensure you have loaded the subject, which must have a retargeted skeleton.
2. In the Subject Setup panel, click the Retargeting tab and expand Constraints.

   

3. In the 3D Scene, select the joint you want to rotate.

4. From the buttons on Subject Setup panel toolbar, select the Enable interactive retargeting button . At the top of the 3D Scene view, the text Interactive Retargeting is displayed.

5. Using the manipulator, rotate the subject's joint.
6. When you're happy with the new position of the joint, release the mouse button. The offsets for the constraints which drive the bone are updated.

Write out a video file of the viewport

With the new playblast command, you can render the active view (3D Scene or Cameras view) to a video or image sequence in Post as a movie file (MOV) so you can share it with other users and use it for reference.

You can use the command either on a take that you manually open and set up, or you can create scripts to automate the process.

For information, see playblast in HSL scripting with Vicon Shogun.

Ability to load static meshes from Shogun Live

In Shogun Post 1.6, you can load an MCP file that was captured in Live and contains a static mesh. When you load the MCP file into Post, the static mesh is displayed as an object whose default translation and rotation values are those that you specified in Shogun Live.

. The default path to which static meshes are saved is:

• C:\Users\Public\Documents\Vicon\SceneMesh\ As with other default locations, you can change the path by editing it in the Preferences dialog box (on the General menu, click Preferences and on the right of the Directories tab, click Scene Meshes).

Export retarget constraints to HSL

The new Export Constraints feature in Shogun Post 1.6 enables you to export your retargeting setup to an HSL file so that you can re-use it. You do this in a similar way to the current process for exporting solving constraints.

To export retargeting constraints and weights:

1. Load a trial with a fully processed subject (VSK and VSS), which has an associated target skeleton (FBX) with a full set of constraints and weights.
2. In the Subject Setup panel, click the Retargeting tab and expand the Constraints section. All the configured constraints and weights are listed.
3. To export constraints, click Export Constraints, which is on the right, underneath the list of constraints.

   

4. In the dialog box, enter or browse to a location and name for the constraints HSL file, and then click Save to export the constraints.
5. To export weights, click Export Weights, which is on the left, underneath the list of constraints.
6. In the dialog box, enter or browse to a location and name for the weights HSL file, and then click Save to export the weights.

To import the exported constraints and weights:

1. Load the MCP file in which you want to use the exported constraints. Ensure that the source bone names that affect constraints and weights are the same as those used in the original MCP file.

   The subject and its solving skeleton are displayed in the 3D Scene view.

2. Load the associated FBX file. Ensure that the target bone names that affect constraints and weights are the same as those used in the original FBX file.

   The FBX target skeleton is displayed in the 3D Scene view.

   Tip: To set the subject's view filter for retargeting, in the 3D Scene view, click View Filters and under Subjects, select Retarget.

3. To import the constraints from the HSL file that you exported earlier, in the Constraints section of the Retargeting tab, click Import Constraints, which is on the right, underneath the constraint list.

4. In the dialog box, enter or browse to the required location and select the constraints HSL file.

5. To import weights, click Import Weights, which ison the left, underneath the constraint list). Note that you must import the constraints first, before you import the weights.

6. In the dialog box, enter or browse to the required location and select the weights HSL file. Ensure the same target root bone is set, and a similar target scale is used as before. The constraints and weights are listed in the Constraints section of the Retargeting tab.

7. Run the usual retargeting process. For more information on retargeting, see Retarget with Shogun Post in Getting started with Vicon Shogun.

New hot keys for retargeting and labeling

Shogun Post 1.6 provides new hot keys for creating constraints when you're retargeting, and for the Labeling panel.

Hot keys for creating constraints

You can now use two new hot keys for creating constraints:

• Ctrl-T Create a retargeting constraint between the selected solving and retargeting bones, or create a solving constraint between the selected solving bone and marker.
• Ctrl-R Create a retargeting rotation constraint between the selected solving and retargeting bones.

For both of these hot keys, the order in which you select the bones and/or markers has no effect.

The default HotKey.hsl file, which now includes the new hot keys, is installed in C:\Program Files\Vicon\ShogunPost1.6\DefaultConfiguration as part of the installation of Post. The two new scripts that the hot keys use (AttachSelectedAsRotationConstraint and AttachSelectedAsPositionConstraint) are also installed. If preferred, you can map custom hot keys to these scripts. For information on creating and using hot keys, see Post - Hot keys and shortcuts in Getting started with Vicon Shogun.

Note that, as with all new hot keys supplied by Vicon, if you want to use the new hot keys, you must copy the default HotKey.hsl file from the above location to C:\Users\Public\Documents\Vicon\ShogunPost1.x, and use it to replace your custom HotKey.hsl file, which is located in this folder. If you do this, any customizations you may have made to the hot keys will be lost, so you may prefer instead to manually merge the two files.

Hot keys for labeling

You can now use hot keys to move up and down the labels list in the Labeling panel.

To do this, in Labeling mode (in the Labeling panel, under Manual Labeling Options, select Label), with the mouse in the view pane and the Labeling panel open, move up and down the list of labels.

• Shift+click to go up through the list
• Shift+right-click to go down through the list

BVH export

With Shogun Post 1.6, you can export takes in BVH format.

If a retargeting skeleton is present, it is exported. If no retargeting skeleton exists, the solving skeleton is exported, if present. As the BVH format permits only one skeleton per file, before you export the scene, ensure that it contains only a single subject/skeleton.

If errors are displayed when you import the resulting BVH, ensure all bones have at least one active DoF before export. (It doesn't matter if they have no keys to go with the DoF(s).)

New and changed HSL scripting commands

Updated commands:

• camerasView New option primarySelectFull enables you to set the Cameras view to show only the selected camera.
• createConstraintsScript: Now has the ability to import and export retargeting constraints and their weights.
• fbxImportOptions: New importToCurrentSubject option enables an FBX to be imported into the current subject hierarchy as a retargeting setup.
• transcodeVideo: Preserves the audio and its sample format by default and has an additional -removeAudio flag to remove the audio.

New commands:

• autoSetupSolvingSkeleton Automates solving skeleton setup.
• autoSetupSolvingSkeletonOptions Sets options for autoSetupSolvingSkeleton.
• getMirrorConstraintCreation: Enables you to restore constraint mirroring after running CreateConstraintsScript.
• playblast: Renders the active view (3D or Cameras) to a video or image sequence.