Tracking in All¹Mix for PTZ cameras via FreeD protocol

General provisions

FreeD – the protocol ensures pan/zoom synchronization of a real PTZ camera and a virtual camera in a 3D scene. This allows All¹Mix to be used as an augmented reality (AR) system. Currently, All¹Mix supports AR mode with Panasonic AW-UE80, AW-UE100 and AW-UE150, Sony BRC-X1000, Lumens VC-A71P cameras.

The image from the PTZ camera is projected onto a special 3D scene object in the "Overlay" mode. Those. texture coordinates are “thrown” from the virtual camera onto this object (Camera Mapping) in each frame, and the diffuse texture of the object’s material is replaced with the video stream from the real camera

There are two main types of using tracking using the FreeD protocol: a) video from a PTZ camera in the background relative to scene objects (augmented reality mode) b) video from a PTZ camera in the foreground relative to scene objects (immersion of real characters/objects into a virtual environment)

Both of these types are implemented using special objects ("Background" and "Foreground"), which All'Mix automatically creates for any 3D scene in the project. Media assigned to these objects is always displayed in "Overlay" mode. Depending on the type of tracking used, the user assigns video from the PTZ camera to “Background” or “Foreground”, respectively. The media assigned to the "Foreground" must of course have a transparency mask (alpha channel).

Hardware setup (using the example of Panasonic AW-UE80 and VC-A71SN)

The camera is configured via the Internet. It is recommended to capture the output stream through input-output cards of the "FD" series (SoftLab-NSK) or "BlackMagic" .

Setting an IP PTZ camera

Connect the camera to the Internet. Camera settings are made in a WEB browser. To do this, you need to type the IP camera in the address bar of your browser. IP cameras and the IP computer on which ALL'MIX is installed must be in the same zone. For different cameras, setting the IP camera is done differently:

1. VC-A71SN has a default IP of 192.168.100.100 On the computer on which All'Mix is installed, you need to change the IP to 192.168.100.101 (for example). Then open your browser and enter the camera's IP address (192.168.100.100) into the address bar. Enter your login and password (default: admin/9999). Change your username and password (optional). Change the network settings of the camera so that the IP of the camera is in the same zone in which the IP of the computer on which All'Mix is installed (DCHP --> off) was originally located.

Save (Apply) the new camera network settings. Return the network settings of the computer on which All'Mix is installed to its original state. Enter the new camera IP address into your browser's address bar. Continue with camera settings.

2. [AW-UE80]. To set an IP camera, use a special utility for Panasonic PTZ cameras: easy-ip-setup-software.

Find the camera (Search), change the network settings of the camera (Network Settings) so that the IP of the camera is in the same zone in which the IP of the computer on which All'Mix is installed was originally located (DCHP --> off). Continue with camera settings (Acces Camera).

! If more than 20 minutes have passed since the selected camera was turned on, you cannot configure the camera. Turn off the camera, then turn it on again. This restriction does not apply to cameras which have not been configured yet or have been reset to their factory default settings.

Setting up tracking in PTZ camera settings

1.VC-A71SN

• In the "Tracking data output (FreeD)" section, uncheck "Enable".
• Assign the IP address of the computer on which All¹Mix is installed and a port (for example 8888).
• Return the checkbox to "Enable".

2.AW-UE80

Tracking Data Output/IP Out

• Set IP(UDP) mode
• Tick at least one client, for example Client 1
• Assign the selected client the IP address of the computer on which All¹Mix is installed
• Assign Port to the selected client, for example 8888
• Press the Set button

Configuring the FD series I/O board

Launch FDConfig2 and set the frame rate to the same as that of the FreeD camera

On all monitors we set the frame rate to the same as that of the FreeD camera.

Setting up tracking in the All¹Mix project

1. Video setup

• add video from a PTZ camera to the project (Media/Live + FD Boards/FD Input HD)
• assign this video to a slot and click on the gear in the title of this slot (Edit input).
• in the media editor window that opens, enable PTZ Controller
• In the properties of PTZ Controller for NDI (recommended), NDI Address is set. If VISCA is selected - IP Address of PTZ camera.

- in the texture properties of a special object (for example, with the name 'Reality') the parameter Tracker on/off should appear

- if Tracker is enabled, then tracking settings appear

2. Setting up a 3D scene

• add a 3D scene to the project (Media/3D Scene + SceneName.fbx)
• assign this scene to a slot and click on the gear in the title of this slot (Edit input).
• in the 3D scene editor window that opens, select the <FOREGROUND> or <BACKGROUND> object or (if it exists in this scene) a previously created special object and assign it as a video source from the PTZ camera.

• Tracker Port - must be the same as in the camera settings
• Tracker Parallax - allows you to compensate for the shift of the optical center when rotating the PTZ camera
• Tracker Offset - changes the initial angular shift between the virtual 3D camera and the real PTZ camera horizontally and vertically
• Tracker Delay - allows you to delay the processing of PTZ coordinates to compensate for the delay required for video processing

Allows you to download the FOV-ZOOM calibration file

Calls up the calibration settings dialog.

Reset tracker calibration - when pressed, a default calibration curve is generated.

3. Calibration dialog

• points on the chart can be edited by dragging
• if you click on any point, the PTZ camera starts sending commands to control the zoom speed until the desired zoom value comes from the camera
• there is a highlighted sliding point on the graph showing the current state of the camera
• graph is saved when saving the project

Creating a special object in a 3ds Max scene

Sometimes more complex geometry of a special object is required. For example, when some scene objects should be in the foreground relative to the character (video from a PTZ camera), and some scene objects should be in the background.

This object must be assigned a material with any original diffuse texture. The material name must begin with a "!". Texture of material with a "!" in All¹Mix is displayed in the "Overlay" mode (in which the texture coordinates of the object are assigned in each frame in a special way (Camera Mapping). If All¹Mix is supposed to assign media with an alpha channel to this material, then the source material must also contain a transparency map. It is recommended to install value 100 for material self-luminosity.

Important: Only those objects whose name (and/or material name) begins with the @ symbol are added to the list of 3D scene objects available for editing. Thus, in our case, the material name of a special object must begin with two characters. For example: !@Reality

For example:

Signs "@", "!" does not appear in the All¹Mix interface. For example, the object name @ObjectName will appear as ObjectName.

Example of working with AR project All¹Mix

Launch the AllMix project Tracking.mixer in presentation mode (F3)

• Assign a camera from the Live FD Boards list to Slot 8
• Go to the Slot 8 editor and enable PTZ Controller
• In the address field, select the desired camera from the drop-down list

Go to the Slot 9 editor, where the 3D scene Tracking.fbx is located

We find a special object (Reality) and assign:

• Input = Slot8
• Tracker = On