WebSocket Communication

WebSocket Communication

Introduction

This is a guide detailing how we are using a WebSocket server to communicate between the Unreal Engine project and the Python Client.

WebSocket Overview

A socket is a connection between two services over a TCP/IP connection. It acts as a programmatic way to send and receive data over the network. Sockets are bidirectional (two-way) and full duplex (can send and receive simultaneously). A socket is composed of an IP address and a port number.

A WebSocket is a network protocol that builds on the idea of sockets. It is built for rapid real-time communication between services, and abstracts away the complexity of managing sockets.

Why WebSockets?

When developing the project, we learned that we needed a way to facilitate communication between the Unreal Engine client and the Python Client. Although Colosseum provides an API that allows the Python Client to gets data from Unreal Engine, there was no way to send data to Unreal Engine. This was a problem because when we provide waypoint overriding from the UI, we need to send data from Unreal Engine to the Python Client.

Our solution is to use a WebSocket server that simply hands sending messages between the Unreal Engine client and the Python client. That way, we can send data both ways.

WebSocket Diagram

This was the approach that the previous team had done for their minimap. They had a Minimap server Python script that sent messages using ROS to a Minimap built within Unity. There are two main differences between our approach and theirs:

  • Our team ported map functionality from Unity to Unreal, eliminating the Unity dependency for the project.
  • Our team is avoiding using ROS to handle data communication due to its setup complexity

Use Cases

As mentioned above, we are using WebSockets to communicate data between the UI and the Python Client. Below is a list of all the cases that require WebSocket communication in the project:

Python Client → UI

  • The Python Client can add a waypoint
  • The Python Client can delete a waypoint
  • The Python Client updates a drone's target in the data table
  • The Python Client updates a drone's state in the data table

UI → Python Client

  • User can add a waypoint through the UI
  • User can delete a waypoint through the UI

JSON Message Structure

Every message sent will contain a MessageType field. The MessageType field contains a string that acts as the identifier for what action to take when receiving the message.

Below is the JSON structure for each of the possible messages that can be sent:

Python Client → UI

  • AddWaypoint:
    
{
    "MessageType": "AddWaypoint",
    "X": float,
    "Y": float,
    "Z": float,
}

  • DeleteWaypoint:
    
{
    "MessageType": "DeleteWaypoint",
    "ID": int,
}

  • UpdateDroneTarget:
    
{
    "MessageType": "UpdateDroneTarget",
    "DroneID": int,
    "WaypointID": int,
}

  • UpdateDroneState:
    
{
    "MessageType": "UpdateDroneState",
    "DroneID": int,
    "State": string
}

UI → Python Client

  • StartSimulation:
    
{
    "MessageType": "StartSimulation",
}

  • StopSimulation:
    
{
    "MessageType": "StopSimulation",
}

  • AddWaypoint:
    
{
    "MessageType": "AddWaypoint",
    "ID": int,
    "X": float,
    "Y": float,
    "Z": float,
    "Priority": int
}

  • DeleteWaypoint:
    
{
    "MessageType": "DeleteWaypoint",
    "ID": int,
}

WebSocket Server

The WebSocket server is implemented in the websocket/websocket_server.py file located in the Drone-Controller repository. This script starts a WebSocket server on ws://localhost:8765

The WebSocket server is started as its own process inside the updated_mission_control.py script with the start_websocket_server() function. This function is the entry point to setting up the WebSocket server. It creates a new asynchronous event loop that handles all async tasks within the WebSocket communication. This new event loop calls the start()

The start() function is what actually starts the WebSocket server. It sets the handler, host, and port values

The handler() function is what a client will perform when it connects to the WebSocket server. When a client sends a message to the server, the server will broadcast the message asynchronously to every other connected client inside the CLIENTS set.

Python Client

The parent_controller.py script handles WebSocket communication on the Python Client side. Inside the perform_startup_sequence() function, a background thread is created to run start_websocket_client_thread().

The start_websocket_client_thread() function, like the WebSocket server, creates a new asynchronous event loop that handles all async tasks within the WebSocket communication. This new event loop calls the connect_to_websocket_server() function.

The connect_to_websocket_server() function is an asynchronous function to connect the client to the server. It then performs an infinite loop of waiting for and receiving messages from the server and putting received messages in the RECEIVED_UI_DATA_QUEUE.

When the Parent Controller enters its perform_startup_sequence() function, it will wait for a StartSimulation message from the UI. This message is sent by the UI client when the user selects a difficulty.

Once the Parent Controller receives the StartSimulation message, it will enter its receive_initial_waypoints() function, where it will process all of the incoming AddWaypoint messages from the UI client.

When the Parent Controller enters its continuous loop() function, each iteration begins by fetching any WebSocket messages in the RECEIVED_UI_DATA_QUEUE and process them inside process_websocket_message()

The process_websocket_message() function will check the MessageType field of the message and perform the appropriate action as defined below:

  • AddWaypoint: Creates a new waypoint and pushes it onto the Waypoint Priority Queue
  • DeleteWaypoint: Deletes an existing waypoint with the given ID from the Waypoint Priority Queue. If a drone's target is deleted, this also sets the drone's current target to None
  • StopSimulation:Fires the SHUTDOWN_EVENT that is shared across all processes

UI Client

The UI client starts with BP_VitalsSimulationGameInstance. This is the GameInstance defined for the project under Project Setting > Maps and Modes. A GameInstance is a high-level manager class for running the game. It contains overridable Init() and Shutdown() functions that are called when the game starts and ends, respectively.

BP_VitalsSimulationGameInstance is derived from the VitalsSimulationGameInstance C++ class. This blueprint only defines three event handlers:

  • An event to get the reference to BP_WaypointManager
  • An event to get the reference to WBP_DroneTable
  • An event to get the reference to WBP_HUD
These events are called by the WebSocketManager class when needed.

VitalsSimulationGameInstance.cpp is the core GameInstance class file. It contains the following:

  • Init(): GameInstance override, creates and initializes the WebSocketManager object
  • Shutdown(): GameInstance override, cleans up the WebSocketManager
It also contains wrapper methods for sending messages. This is so that other blueprints can call these methods once they get a reference to the GameInstance class.

The WebSocketManager.cpp class is what handles all WebSocket communication. It is initialized within VitalsSimulationGameInstance.cpp as described above. Below is a list of the public methods that are called by the methods of the GameInstance:

  • Initialize(): Continuously probe for a connection to the server. Once connected, it sets up the WebSocket Event Handlers
  • Close(): Closes the client connection
  • SendAddWaypointMessage(): Sends an AddWaypoint message to the server
  • SendDeleteWaypointMessage(): Sends a DeleteWaypoint message to the server
  • SendStartSimulationMessage(): Sends a StartSimulation message to the server
  • SendStopSimulationMessage(): Sends a StopSimulation message to the server

When a message is received from the WebSocket server, the message is parsed as a JSON object and is sent to the appropriate handler function based on the MessageType field.

The JSON module is a required dependency for Unreal Engine to parse JSON messages. The JSON module is included as a dependency in the DroneEnvironment.Build.cs file.

All message handlers in WebSocketManager are set up so that they call some other event in a specific blueprint. In other words, they are calling a Blueprint-defined event from a C++ class. In general, they perform the following operations:

  1. Get the associated data from the JSON message
  2. Get a function pointer to the event from the Blueprint you want to call
  3. Create a struct containing the parameters for the event
  4. Call ProcessEvent() with the function pointer and the parameters as arguments

For example, this is the general outline for when the UI client receives an AddWaypoint message from the WebSocket server:

  1. Get the X, Y, and Z values from the JSON message
  2. Get a function pointer to the AddWaypointEvent found in BP_WaypointManager
  3. Create a struct that matches the AddWaypointEvent parameters
  4. Call ProcessEvent with the function pointer and parameter struct as arguments
This will fire the AddWaypointEvent event in the BP_WaypointManager blueprint