Batch process files in Shogun Post
- Adaptavist ScriptRunner for Confluence Cloud (Unlicensed)
Batch Processing in Shogun Post enables a specific operation to be performed on a group of files. Operations can range from reprocessing, file export, custom pipelines, or scripts. Generally, an operation opens a file, performs a process that you have defined, and exports a new file (however, not all of those steps are required and it depends on your selected process).
Shogun Post can batch process files sequentially in the application or in parallel—the latter enables you to continue with other tasks while batch processing continues in the background. You can process files in parallel within a single PC, or “farm” out the work across multiple PCs.
The following examples explain different batch processing methods and when to use them.
Batch process locally within Shogun Post
For simple processing jobs, one approach is to process the files on a single machine. For example, you may have 100 files that are all cleaned up, ready to export, and the exporting process takes less than a minute per file. In this case, it’s much easier to process the files on one machine (as it only uses one core of your PC’s CPU). You can do other tasks in another instance of Shogun Post while the files are being processed. However, you can achieve the same result quicker by batch processing the jobs in parallel, or by distributing the work (also known as “farming”). For more information, see Set up distributed batch processing.
For simple processing jobs, load your files into the Batching panel, configure the processing settings (in either the Batching or Processing panel), and start processing your files. The smaller the file size, the quicker the processing time.
Batch process files in parallel using a single PC
Some processing operations are limited to a single or small number of cores, most often to achieve better processing quality. If you use this approach, we recommend you use a machine with the highest amount of cores available, so the processing can be done on a single machine using these different cores. To use this single-machine approach, run multiple instances of the ShogunPostCL workers on the same machine as Shogun Post controller, and then farm out the processing job across the ShogunPostCL workers. For more information on running workers on the same machine as the controller, see Set up workers locally.
Batch process files in parallel using multiple PCs
Use a single PC to run Shogun Post as a controller and farm out the work by running ShogunPostCL on other machines to act as workers. Each file to be processed is then sent to a worker that opens the file, performs the required processing, and saves the processed file as a new file.
Distribute reconstruction of a single file across multiple cores or machines
For better reconstruction quality, use a low number of cores when processing the files. Longer takes require more time to reconstruct, however, you can use an option to split the take into time ranges so you can distribute the reconstruction for each of these time ranges.
Set up local processing
If you need to process one or two moderately sized files (less than 1 GB) or have lots of small files to process, it is easier to process these files on just one machine. To do so, follow these steps:
In Shogun Post, on the Panels tab of the ribbon, select Batching.
From the Mode list, select one of the batching modes:
Reconstruct
Post Process
Process Video
Replace Subjects
Custom
Configure the options. The mode that you select determines the options available to configure.
In the Batching panel, expand the Farm section and make sure Farm is not selected. (Farming is only required for parallel batch processing).
Ensure the Files and Processing section is expanded and click Add Files to select the files to process.
After you have added the files, click Start.
When processing locally, the progress of each file in the batch operation is displayed in the lower left of the Batching panel.
Set up distributed batch processing
To set up batch processing, first set up your workers and then set up the controller to set up the distribution.
You can set up workers in either of these ways:
Set up workers across multiple machines
Make sure all the machines are using the same version of Shogun Post. To find the version number, go to Help > About Shogun Post, and note the version number that is displayed in the dialog box.
Make sure each machine can access the input materials, such as the files to batch, calibrations, and custom scripts.
Tip: Place these files in a shared network location so each machine can point to the same input material source.Launch ShogunPostCL.exe on each machine. The .exe file is found in C:\Program Files\Vicon\ShogunPost1.1#.
In each instance of ShogunPostCL, enter
slave on.
ShogunPostCL is now in batch worker mode and is ready to receive commands from the controller (Shogun Post).
Set up workers locally
Launch as many instances of ShogunPostCL as required (The .exe file is found in C:\Program Files\Vicon\ShogunPost1.1#). The number of instances needed depends on the type of processing you require and the resources of the local machine (such as the number of cores or available memory).
In each instance of ShogunPostCL, enter
slave on.
The workers are now in batch worker mode and are ready to receive commands from the controller (Shogun Post).
Set up distribution in Shogun Post
After you have set up the workers, set up the batch processing distribution using a controller (Shogun Post).
In Shogun Post, on the Panels tab of the ribbon, select Batching.
From the Mode list, select one of the batching modes:
Reconstruct
Post Process
Process Video
Replace Subjects
Custom
Configure the options. The mode that you select determines the options available to configure.
If you are processing across multiple machines, ensure the file paths to the input material are specified and the location is accessible to each worker machine.
In the Batching panel, expand the Farm section and select Farm.
A list of workers is displayed in the Farm section. You can add and remove workers by selecting Enabled for the relevant worker.
Expand the Files and Processing section and click Add Files to select the files to process.
After you have added the files, click Start.
A log displays the file status along with the worker that is processing each file.