This article is an exploration of the technologies used in developing an MMO Server Architecture that is robust, modular, scalable and above all, high performance. The core library is designed around a plugin architecture that allows developers to extend the engine’s functionality and add custom features.

Here are some typical components of a plugin architecture:

1. Plugin Interface/Contract: A well-defined interface or contract that plugins must adhere to. It defines the methods, events, or hooks that plugins can implement or subscribe to. The interface provides a standardized way for the core system to interact with plugins and vice versa.
2. Plugin Manager: The plugin manager is responsible for discovering, loading, initializing, and managing plugins. It provides functionality to scan designated directories, locate plugin files, and dynamically load them into memory. The plugin manager also handles versioning, dependency management, and lifecycle management of plugins.
3. Plugin Loader: The plugin loader is responsible for dynamically loading the plugin code into the system’s runtime environment. It performs tasks such as loading the plugin assembly or dynamically linking the plugin code at runtime. The loader handles the loading process, including security considerations, assembly isolation, and error handling.
4. Plugin Configuration: A mechanism for configuring and customizing the behavior of individual plugins. This may involve providing configuration files or settings that plugins can read at runtime. The configuration allows users or administrators to enable/disable specific plugins, set parameters, or define plugin-specific behavior.
5. Plugin Events and Hooks: Events or hooks provided by the core system that plugins can subscribe to. These events are typically triggered at various points during the system’s execution, allowing plugins to extend or modify the system’s behavior. Plugins can register event handlers or hooks to respond to specific events and perform custom logic.
6. Plugin Communication/Message Passing: Mechanisms for communication and data exchange between plugins and the core system. This can include passing messages, using shared memory, or utilizing event-based systems like publish-subscribe or message queues. Communication mechanisms enable plugins to collaborate, share information, or perform coordinated actions.
7. Plugin Lifecycle Management: Facilities for managing the lifecycle of plugins, including plugin activation, deactivation, installation, uninstallation, and updates. The plugin lifecycle management ensures that plugins can be safely loaded, initialized, and unloaded without disrupting the core system’s stability or functionality.
8. Plugin Security and Sandboxing: Security measures to protect the core system from potentially malicious or poorly behaving plugins. This may involve sandboxing plugins, applying permission-based access controls, or enforcing code verification mechanisms. The security measures help ensure that plugins cannot compromise the integrity or security of the system.

Plugin Manager

Let’s look at the interface for Plugin Manager.

Plugins are loaded from shared libraries. When the library is loaded the Plugins register with the Plugin Manager.

class IPluginManager
{
	virtual bool loadPluginConfig();

	virtual bool loadPlugins();

	virtual bool onAwake();

	virtual bool onInit();

	virtual bool onPreUpdate();		

	virtual bool onUpdate();

	virtual bool onPostUpdate();

	virtual bool onBeginShutdown();

	virtual bool onShutdown();
	
	virtual bool onEndShutdown();
	
	virtual void registerPlugin(IPlugin* plugin) = 0;

	virtual void unRegisterPlugin(IPlugin* plugin) = 0;

	virtual IPlugin* findPlugin(const std::string& pluginName) = 0;

};

Plugins

Plugins are mostly like wrappers around a shared library. The plugin is used as a standard interface to add new runtime functionality in the form of Modules. When a Plugin is first loaded it creates and registers all of its Modules .

Below is the interface for Plugin.

class IPlugin  
{  
 virtual const int getPluginVersion() = 0; 
 virtual const std::string& getPluginName() = 0;  
 virtual void registerModules() = 0;  
 virtual void unregisterModules() = 0; 
 virtual void addModule(const std::string& name, IModule* module); 
 virtual void IModule* getModule(const std::string& name); 
 virtual void removeModule(const std::string& name); 
 virtual bool onAwake(); 
 virtual bool onInit(); 
 virtual bool onPostInit(); 
 virtual bool onPreUpdate(); 
 virtual bool onUpdate(); 
 virtual bool onPostUpdate(); 
 virtual bool onBeginShutdown(); 
 virtual bool onShutdown(); 
 virtual bool onEndShutdown();  
};

Below are some example of Modules:

1. Actor Module: The Actor Module is responsible for encapsulating the functionality for creating and destroying game objects, as well as registering and adding components to them which will define their behavior.
2. Communication Module: Handles network communication between the server and clients. It can utilize technologies like IOCP (Input/Output Completion Ports) for high-performance and efficient I/O operations.
3. Property Module: The Property Module provides an interface for accessing and manipulating data in a modular and extensible way. 
4. Game Logic Module: Implements the game rules, mechanics, and simulation. It ensures consistency and fairness across all clients by validating player actions and enforcing game rules.
5. Database Module: Manages the persistent data of the MMO, such as player profiles, inventories, and world state. It interacts with a database system to store and retrieve data efficiently.
6. Scene Module: Loads and manages scenes and scene data. Provides callbacks to events like onEnter() and onExit().
7. AI Module: Handles non-player character (NPC) behavior and artificial intelligence. It controls the actions and decision-making of NPCs in the game world.
8. Navigation Module: Generates and maintains live nav mesh data, can be used to offload pathing, nav mesh generation, etc.
9. Kernel Module:  The Kernel Module provides low-level functionality and interacts directly with the hardware and system resources. This can include managing processes and threads, memory allocation and management, or handling inter-process communication.
10. Security Module: Implements security measures to protect sensitive player data and prevent cheating. This module can include authentication, encryption, and anti-cheat mechanisms.

What’s a Property Module?

A Property Module is a software architecture that provides an interface for accessing and manipulating data in a modular and extensible way. Properties are wrappers around a value that provide an easy-to-use interface for connecting them to arbitrary callbacks. A property container can store arbitrary properties and provides a way to organize and access properties within the context of a plugin or module. The specific implementation and features of a property system can vary depending on the framework or library being used.

Some features of a modern C++ property system for storing data in a plugin architecture include:

1. Property Wrappers: Properties are wrappers around a value that provide an easy-to-use interface for accessing and manipulating the underlying data. They encapsulate the data and provide getter and setter methods to interact with it.
2. Callback Support: Properties can be connected to arbitrary callbacks or event handlers. This allows for additional logic to be executed whenever the property value changes or is accessed.
3. Property Containers: A property container is a data structure that can store and manage multiple properties. It provides a way to organize and access properties within the context of a plugin or module.
4. Type Safety: The property system should ensure type safety, meaning that properties can only be accessed and manipulated in a way that is consistent with their declared type. This helps prevent type-related errors and ensures data integrity.
5. Reflection and Introspection: A property system can include reflection capabilities, allowing for runtime inspection of properties and their metadata. This can be useful for dynamically discovering and manipulating properties at runtime.
6. Serialization and Persistence: The property system includes mechanisms for serializing and persisting property data. This allows for saving and loading the state of properties, enabling data persistence across sessions or for sharing with other modules or plugins.

An Simple C++ Example:

IObject* pObject = new GameObject(GET_GUID(0, 1), pluginManager);

pObject->GetPropertyManager()->AddProperty(pObject->Self(), "Server Name", DT_STRING);
pObject->GetPropertyManager()->AddProperty(pObject->Self(), "World Count", DT_INT);


pObject->SetPropertyString("Server Name", "Heroes Province Game Server");
pObject->SetPropertyInt("World Count", 10);

int n1 = pObject->GetPropertyInt32("World Count");
std::cout << "World Count:" << n1 << std::endl;

Communication Module

The Communication Module is a specially designed SDK to accelerate network traffic and detect optimal routes in real-time. This involves analyzing network traffic data, identifying bottlenecks, and making routing decisions to optimize the flow of data. In addition, the Communication Module provides access to all low level networking, messaging, and Iow latency, non blocking, I/O operations. This is the base interface for server communication:

///Server
class ServerCommunicationModule : public IModule
{
NetServer* onServerCreate(void* context, const char* server_address, const char* bind_address, const char* datacenter, void (*packet_received_callback)(NetServer* server, void* context, const NetAddress* from, const uint8_t* packet_data, int packet_bytes));

void onServerDestroy(NetServer* server);

uint16_t getServerPort(NetServer* server);

NetAddress getServerAddress(NetServer* server);

int getServerState(NetServer* server);

void onServerUpdate(NetServer* server);

void onServerSendPacket(NetServer* server, const NetAddress* to_address, const uint8_t* packet_data, int packet_bytes);

std::string getServerStats(NetServer* server, const NetAddress* address);

bool onServerReady(NetServer* server);

void onServerEvent(NetServer* server, const NetAddress* address, uint64_t server_events);

void onServerMatch(NetServer* server, const NetAddress* address, const char* match_id, const double* match_values, int num_match_values);

void onServerFlush(NetServer* server);
};

And here is the base interface for Client Communication

/// Client
class ClientCommunicationModule: public IModule
{
NetClient* onClientCreate(void* context, const char* bind_address, void (*packet_received_callback)(NetClient* client, void* context, const NetAddress* from, const uint8_t* packet_data, int packet_bytes));

void onCientDestroy(NetClient* client);

uint16_t getClientPort(NetClient* client);

void onClientStartSession(NetClient* client, const char* server_address);

void onClientStopSession(NetClient* client);

bool getClientIsReady(NetClient* client);

int getClientState(NetClient* client);

void onClientUpdate(NetClient* client);

void onClientSendPacket(NetClient* client, const uint8_t* packet_data, int packet_bytes);

std::string getClientStats(NetClient* client);

const NetAddress* getClientServerAddress(NetClient* client);	
};

What are IOCP (Input /Output Completion Ports)?

IOCP (Input/Output Completion Ports) is a programming model and technology used in Windows operating systems to efficiently handle asynchronous I/O operations. It provides a scalable and high-performance solution for managing I/O operations, particularly in server applications that handle a large number of concurrent connections, such as MMO servers 8, 9, 10.

Here’s how IOCP works:

1. Completion Ports: A completion port is a kernel-level I/O management facility provided by the Windows operating system. It acts as a queue where I/O completion packets are placed when asynchronous I/O operations complete.
2. I/O Completion Packet: An I/O completion packet contains information about a completed I/O operation, such as the I/O status, the number of bytes transferred, and the associated data. When an asynchronous I/O operation finishes, the I/O subsystem generates an I/O completion packet and places it in the completion port’s queue.
3. I/O Overlapped Operations: In an application using IOCP, I/O operations are typically performed asynchronously using overlapped I/O. When an I/O operation is initiated, the application provides an overlapped structure that contains information about the operation, including the buffer for data transfer and a callback function to be invoked when the operation completes.
4. I/O Completion Notifications: The application associates a socket or file handle with an IOCP object using a function call. This indicates that I/O operations on that handle should be monitored by the IOCP. When an I/O operation on a monitored handle completes, the operating system generates an I/O completion packet and places it in the associated completion port’s queue.
5. IOCP Thread Pool: The application creates a pool of threads that are dedicated to handling completed I/O operations. These threads retrieve completion packets from the completion port’s queue, extract the relevant information, and execute the associated callback function or perform the necessary processing.

By using IOCP, an application can efficiently manage a large number of asynchronous I/O operations with a small number of threads. The IOCP model provides several benefits, including reduced resource consumption, efficient handling of concurrent I/O operations, and improved scalability for server applications that need to handle a large number of connections simultaneously.

Scalable Infrastructures

Modern MMO’s employ a combination of load balancing, server clustering, hybrid architectures, scalable infrastructure, and performance optimization techniques to handle server load and ensure scalability. These strategies allow the games to accommodate a large number of concurrent players and provide a smooth and enjoyable gaming experience.

Here are some common approaches:

1. Load Balancing: MMO games often employ load balancing techniques to distribute the player load across multiple servers. This helps prevent any single server from becoming overwhelmed and ensures that the game can handle a large number of concurrent players. Load balancing can be achieved by dividing the game world into separate geographical areas, with each area being handled by a dedicated server 1. This allows the game to scale horizontally by adding more servers as needed.
2. Server Clustering: In some cases, MMO games use server clustering to achieve scalability. Clustering involves grouping multiple servers together to work as a single unit. This allows the game to distribute the load across the cluster and handle more players. Clustering can also provide redundancy and fault tolerance, as if one server fails, the workload can be automatically shifted to other servers in the cluster 5.
3. Hybrid Architectures: MMO games may employ hybrid architectures that combine client-server and peer-to-peer (P2P) systems. This approach allows the game to offload some of the processing and networking tasks to the players’ machines, reducing the load on the central servers. P2P systems can be used for tasks such as player movement and non-critical game events, while critical game logic and data are handled by the central servers 3.
4. Scalable Infrastructure: MMO games often rely on scalable infrastructure, such as cloud computing platforms, to handle server load and scalability. Cloud platforms provide the ability to dynamically allocate resources based on demand, allowing the game to scale up or down as needed. This flexibility ensures that the game can handle peak loads during busy periods and scale back during quieter times 4.
5. Performance Optimization: MMO games also employ various performance optimization techniques to improve server efficiency and reduce the load. This includes optimizing network protocols, minimizing bandwidth usage, and implementing caching mechanisms. By optimizing performance, MMO games can handle more players with fewer server resources 4.

High level server architecture

Master Server

-Manages all login servers and World Servers

-Can be configured to use multiple Master Servers to avoid a single point of failure.

This is the interface for the Master Server Module:

class IMasterServerModule : IModule
{

virtual std::string getServersStatus() const;


void OnHeartBeat(const int msgID, const char* msg, const uint32_t len);
void onValidateMessage(const int msgID, const char* msg, const uint32_t len);


void OnMasterServerRegister(const int msgID, const char* msg, const uint32_t len);
void OnMasterServerUnRegister(const int msgID, const char* msg, const uint32_t len);

void OnLoginRegister(const int msgID, const char* msg, const uint32_t len);
void OnLoginUnRegister(const int msgID, const char* msg, const uint32_t len);

void OnWorldRegister(const int msgID, const char* msg, const uint32_t len);
void OnWorldUnRegister(const int msgID, const char* msg, const uint32_t len);

void OnGameServerRegister(const int msgID, const char* msg, const uint32_t len);
void OnGameServerUnRegister(const int msgID, const char* msg, const uint32_t len);
	
void OnSelectWorldServer(const int msgID, const char* msg, const uint32_t len);
	
void OnServerReport(const int msgId, const char* buffer, const uint32_t len);

protected:

std::unordered_map<ServerId, ServerData> mMasterServers;
std::unordered_map<ServerId, ServerData> mLoginServers;
std::unordered_map<ServerId, ServerData> mWorldServers;
std::unordered_map<ServerId, ServerData> mProxyServers;
std::unordered_map<ServerId, ServerData> mGameServers;

};

Each Master Server tracks all other Master, Login, World, Proxy and Game Server.

World Server

-Manages the Game Servers and Proxy Servers

-Can be configured to use multiple Master Servers to avoid a single point of failure.

This is the interface for the World Server

class IWorldServerModule: public IModule
{

virtual int getWorldServerId() = 0;
virtual int getWorldAreaId() = 0;

virtual void processMessages(const int msgID, const char* msg, const uint32_t len) = 0;

virtual bool sendMsgToGame(const int gameID, const int msgID, const std::string& data) = 0;
virtual bool sendMsgToGame(const int gameID, const int msgID, const google::protobuf::Message& data) = 0;

virtual bool SendMsgToGamePlayer(const PlayerId nPlayer, const int msgID, const std::string& data) = 0;
virtual bool SendMsgToGamePlayer(const PlayerId nPlayer, const int msgID, const google::protobuf::Message& data) = 0;

virtual const std::vector<PlayerId>& getPlayers() = 0;
virtual std::shared_ptr<PlayerData> getPlayerData(const int id) = 0;

};

Login Server

Login and Authentication Server: This server manages player authentication, account creation, and login processes. It verifies player credentials, handles secure communication, and generates access tokens or session IDs that allow players to connect to the game servers.

Game Server

The game server is the core component that handles the game logic, simulation, and communication between players. It manages the game world, non-player characters (NPCs), items, quests, and other gameplay elements. Game servers handle player input, update the game state, and broadcast relevant information to connected clients.

Proxy Server

MMO architectures often employ proxy or gateway servers as intermediaries between the players and the game servers. These servers handle network traffic, load balancing, and routing of player connections to different game servers based on player location, server availability, and server load. They help distribute the player load across multiple servers and improve overall scalability.

Database Server

MMOs often rely on database servers to store and manage persistent game data, such as player profiles, inventory, achievements, and other player-related information. Database servers are responsible for handling read and write operations efficiently, ensuring data consistency, and scaling to handle large volumes of data.

Chat Server

The chat server facilitates real-time communication, including global, regional, and private chat channels. It relays messages between players, handles filtering and moderation, and manages chat-related features such as emotes and chat commands.

Backend Services

MMOs often require additional backend services for various purposes, such as billing and microtransactions, player analytics, customer support, server monitoring, and maintenance. These services provide administrative tools, gather data on player behavior, handle in-game purchases, and ensure the smooth operation of the game.

Load Balancing

Sharding in MMO servers refers to the practice of dividing the game’s universe or world into separate instances or shards to manage player populations and server issues 2. Here is a description of how MMO servers handle sharding:

1. Player Sorting: Within the data center, players are sorted into “home worlds” or shards 3. This sorting process ensures that players are grouped together based on various factors such as region, language, or player preferences.
2. Geographical Division: One common approach to sharding is to divide the game world into many separate geographical areas 4. Each geographical area is handled by a dedicated server, which is responsible for managing the game state and player interactions within that specific area.
3. Load Balancing: Sharding helps distribute the player load across multiple servers, allowing the game to handle a larger number of concurrent players. Load balancing techniques are often employed to ensure that the player population is evenly distributed across the shards 4.
4. Shard Transfer: Depending on the game, shard transfer boundaries can be established to allow players to move between shards or instances. This allows players to join their friends or access specific content in different shards1.
5. Single-Shard Architecture: Some MMO games, like EVE Online, adopt a single-shard architecture where all players exist in the same game universe 2. This approach eliminates the need for sharding and allows for a seamless and unified experience. However, running an MMO in a single shard introduces challenges in system architecture, runtime, databases, and operations 2.

Implementing horizontal sharding in MMO games can present some challenges. Here are some common challenges:

1. Data Consistency: Horizontal sharding can make it challenging to maintain data consistency across different servers. If a player’s data is split across multiple servers, it can be difficult to ensure that all the data is updated correctly and consistently1.
2. Query Complexity: Horizontal sharding can lead to more complex queries, as data may need to be retrieved from multiple servers to answer a single query. This can impact query performance and increase the workload on the servers6.
3. Load Balancing: Horizontal sharding can make load balancing more challenging, as the player population may not be evenly distributed across the shards. This can lead to some servers being overloaded while others are underutilized1.
4. Shard Transfer: Depending on the game, shard transfer boundaries can be established to allow players to move between shards or instances. However, this can be challenging to implement in a horizontally sharded environment, as players may need to transfer data between servers3.
5. Maintenance and Operations: Horizontal sharding can make maintenance and operations more complex, as there are more servers to manage and maintain. This can increase the workload on the operations team and impact the game’s availability6.

Here’s how Horizontal Sharding fits into our high-level server architecture:

Shard Manager:

• The shard manager is responsible for creating, managing, and load balancing shards within the MMO server architecture.
• It monitors the player population and the resource utilization of each shard to determine when to create new shards or merge existing ones based on predefined criteria, such as player density or server capacity.

Shard Instances:

• Each shard consists of one or more server instances, each running a copy of the game world and managing a subset of players.
• Shard instances handle game logic, player interactions, NPC behavior, and other gameplay-related processes.
• The number of shard instances within a shard can vary based on factors like the expected player population and server performance requirements.

Inter-Shard Communication:

• Shards may need to communicate with each other in certain scenarios, such as when players from different shards interact or during cross-shard events.
• Inter-shard communication mechanisms, such as message queues, distributed databases, or dedicated communication channels, enable seamless interactions and data exchange between shards when necessary.

Shard-Specific Data:

• Each shard may have its own set of shard-specific data, such as persistent game data, player profiles, and shard-specific settings.
• Shard-specific data is stored in separate databases or file systems to ensure isolation and maintain the integrity of individual shards.

Continue this Blog in Modern Server Architecture Part 2