Large Model Game Interaction Access (Android)

In games that support AI interaction, users can interact with large model AI players through voice or text. The premise for interaction is that there must be at least one real person in the game, and then the large model AI player can be added through the corresponding interface. Subsequent interactions can be conducted through the ISudAiAgent-related interfaces.

ISudAiAgent Interface

public interface ISudAiAgent {
    /**
     * For voice conversations, send audio data through this interface, not required to implement.
     * The audio slice passed in is PCM data obtained from RTC or system recording.
     * The PCM data format must be: sample rate: 16000, bit depth: 16, number of channels: MONO.
     * The PCM data length can be adjusted according to the effect; longer length: better accuracy but longer delay; shorter length: shorter delay but sacrifices accuracy.
     */
    void pushAudio(ByteBuffer data, int dataLength);
    /** Send text content */
    void sendText(String text);
    /** Pause voice recognition, temporarily stop calling pushAudio */
    void pauseAudio();
    /** Stop voice recognition */
    void stopAudio();
    /** Set callback */
    void setISudListenerAiAgent(ISudListenerAiAgent listener);
}

I. Obtaining ISudAiAgent

After loading the game, obtain ISudAiAgent through the ISudFSTAPP interface method getAiAgent. In the demo, you can use the method getAiAgent defined in BaseGameViewModel to obtain ISudAiAgent. Once the ISudAiAgent interface is obtained, you can interact with the large model AI player via voice and text and set listeners to receive replies from the large model AI player.

Usage Example:

public class QuickStartGameViewModel extends BaseGameViewModel {
    @Override
    public void onGameStarted() {
        super.onGameStarted();
        // Ensure the returned object is not null after onGameStarted is triggered
        ISudAiAgent aiAgent = getAiAgent();
    }
}

II. Adding Large Model AI Players

Interface definition document: General Status notifyStateChange

52. Set large model AI players in the game
app_common_game_add_big_scale_model_ai_players

When there is at least one real person in the game, the client can send a message to the game in the protocol format defined by this interface to add large model AI players.

// In Demo/QuickStartActivity, you can call this to add large model AI players
SudGIPAPPState.APPCommonGameAddBigScaleModelAIPlayers aiPlayers = new SudGIPAPPState.APPCommonGameAddBigScaleModelAIPlayers();
aiPlayers.aiPlayers = new ArrayList<>();
SudGIPAPPState.ModelAIPlayers modelAIPlayers = new SudGIPAPPState.ModelAIPlayers();
modelAIPlayers.userId = "userId";
modelAIPlayers.avatar = "avatar";
modelAIPlayers.name = "name";
modelAIPlayers.gender = "male";
modelAIPlayers.aiIdStr = "1";
aiPlayers.aiPlayers.add(modelAIPlayers); // You can set one or more at a time
gameViewModel.notifyStateChange(SudGIPAPPState.APP_COMMON_GAME_ADD_BIG_SCALE_MODEL_AI_PLAYERS, aiPlayers);

III. Text Dialogue

The app can choose to send text messages to the game to interact with the large model AI player. Use the ISudAiAgent interface method sendText to send text content to the game.

public interface ISudAiAgent {
    ...
    /** Send text content */
    void sendText(String text);
    ...
}
// In Demo/QuickStartActivity, you can call this to send text messages to the game
ISudAiAgent aiAgent = gameViewModel.getAiAgent();
if (aiAgent != null) {
    aiAgent.sendText("text");
}

IV. Voice Dialogue

The app can choose to send voice messages to the game to interact with the large model AI player. Use the ISudAiAgent interface method pushAudio to send voice messages to the game. pauseAudio is used to pause voice dialogue, and stopAudio is used to stop voice dialogue.

public interface ISudAiAgent {
    ...
    /**
     * For voice conversations, send audio data through this interface, not required to implement.
     * The audio slice passed in is PCM data obtained from RTC or system recording.
     * The PCM data format must be: sample rate: 16000, bit depth: 16, number of channels: MONO.
     * The PCM data length can be adjusted according to the effect; longer length: better accuracy but longer delay; shorter length: shorter delay but sacrifices accuracy.
     */
    void pushAudio(ByteBuffer data, int dataLength);
    /** Pause voice recognition, temporarily stop calling pushAudio */
    void pauseAudio();
    /** Stop voice recognition */
    void stopAudio();
    ...
}
// In Demo/QuickStartActivity, you can call this to send voice data to the game
ISudAiAgent aiAgent = gameViewModel.getAiAgent();
if (aiAgent != null) {
    aiAgent.pushAudio(data, dataLength);
}

V. Receiving Messages from Large Model AI Players

When the app sends messages to the large model AI player through voice or text, the large model AI player will reply with one or more messages to the players in the room. Use the ISudAiAgent interface method setISudListenerAiAgent to set a listener for the replies from the large model AI player.

public interface ISudAiAgent {
    ...
    /** Set callback */
    void setISudListenerAiAgent(ISudListenerAiAgent listener);
    ...
}
public interface ISudListenerAiAgent {
    /** Large model AI message */
    void onRoomChatMessage(String json);
}

// JSON format returned by ISudListenerAiAgent.onRoomChatMessage
{
    "uid": "26", // UID of the large model AI player
    "content": "I am a Gemini, what about you?", // Text content spoken by the large model AI player, corresponds to the audio
    "audioData": "base64 formatted audio data" // Voice content spoken by the large model AI player, corresponds to the text
}

// In Demo/QuickStartGameViewModel, you can call this to listen for messages sent by the large model AI player
public class QuickStartGameViewModel extends BaseGameViewModel {
    @Override
    public void onGameStarted() {
        super.onGameStarted();
        // Ensure the returned object is not null after onGameStarted is triggered
        ISudAiAgent aiAgent = getAiAgent();
        aiAgent.setISudListenerAiAgent(new ISudListenerAiAgent() {
            @Override
            public void onRoomChatMessage(String json) {
                try {
                    JSONObject obj = new JSONObject(json);
                    String audioData = obj.getString("audioData"); // Base64 formatted audio data, decode it to get the mp3 binary audio data
                    String uid = obj.getString("uid"); // UID of the large model AI player
                    String content = obj.getString("content"); // Text content, corresponds to the spoken content, decide whether to display based on your product needs
                    playAudioData(uid, audioData);
                } catch (Exception e) {
                    e.printStackTrace();
                }                
            }
        });
    }
}

When the onRoomChatMessage callback is received, the spoken content of the large model AI player can be displayed as text messages or played as voice messages according to the app's product needs.

VI. Voice Wave Control

When the app receives the following game message, it indicates that the app can choose to control the display of the corresponding player's voice wave effect in the game.

Interface definition document: General Status - Game

69. Notify the app that it can start pushing the microphone speaking status
mg_common_game_player_mic_state

After receiving the instruction, you can control it through the following interface.

Interface definition document: General Status notifyStateChange

51. App sends player mic state to the game
app_common_game_player_mic_state

This is used to control whether to display the specified player's voice wave in the game, and it is a toggle-type instruction; it only needs to be sent once when enabling or disabling.

Reference Example:

```java // In Demo/QuickStartGameViewModel, you can call this to control the display of voice waves in the game public class QuickStartGameViewModel extends BaseGameViewModel { @Override public void onGameMGCommonGamePlayerMicState(ISudFSMStateHandle handle, SudGIPMGState.MGCommonGamePlayerMicState model) { super.onGameMGCommonGamePlayerMicState(handle, model); // After receiving this callback, you can control the display of voice waves as shown in the following example SudGIPAPPState.AppCommonGamePlayerMicState micState = new SudGIPAPPState.AppCommonGamePlayerMicState(); micState.uid = "userId"; micState.state = 1; // 0: Stop speaking, 1: Speaking notifyStateChange(SudGIPAPPState.APP_COMMON_GAME_PLAYER_MIC_STATE, micState); } }

Android