miHoYo Unveils 'Gamer,' an Open-Weight AI Agent That Plays Honkai: Star Rail via Screen Pixels and Mouse Inputs
miHoYo’s in-house Honkai AI R&D Team, working alongside researchers from Tsinghua University, the National University of Singapore, the University of Hong Kong, and the University of Chinese Academy of Sciences, has unveiled **Gamer**, an open-weight Vision-Language-Action foundation model capable of playing complex 3D and 2D games—including *Honkai: Star Rail*—from start to finish using only raw screen pixels and standard mouse and keyboard inputs. Published on arXiv on November 13, 2026, and reported across Japanese specialist press by [Automaton West](https://automaton-media.com/en/news/mihoyo-develops-ai-model-that-can-play-complex-games-like-honkai-star-rail-just-like-a-human-would-with-mouse-and-keyboard-inputs/) on November 18, the project stands out because it touches zero lines of backend game code. Unlike conventional game bots that read internal memory states, object coordinates, or studio-exposed APIs, Gamer operates strictly through the same visual and physical loop available to human players: it ingests 5 Hz screen captures and emits mouse displacements, scroll events, and keyboard state blocks. ### Breaking the Real-Time Latency Wall Why have general-purpose multimodal models struggled so badly in live action games? On paper, frontier vision-language models already score well on static screenshots and turn-based reasoning. In practice, however, benchmarks published on the [Gamer project page](https://mihoyo-gamer.github.io/) show that switching from a "time-paused" evaluation—where the game engine freezes while the model thinks—to real-time execution causes general-purpose models to lose 17 to 35 percentage points on average across suites like MCU, ViZDoom, OmniGameArena, and GameWorld. By the time a standard cloud LLM finishes decoding a multi-token response, the enemy has moved or the timing window has closed. Gamer resolves that bottleneck across both its action representation and its serving stack. Rather than emitting one coarse action per screenshot or relying on a separate continuous control head, the model takes one screenshot every 200 milliseconds (5 Hz) and autoregressively predicts a mouse displacement vector alongside six consecutive 33-millisecond key-state blocks. That design yields an effective 30 Hz control cadence directly inside the language model’s token stream. To keep inference inside that 200 ms budget, the team built **Streaming-vLLM**, an optimized serving engine hosted alongside the [official GitHub repository](https://github.com/RainBowLuoCS/Gamer). By pinning each client’s KV cache across turns, deploying architecture-specific kernels, and dispatching verified action blocks early before the full response finishes decoding, Streaming-vLLM slashes end-to-end request latency on an NVIDIA H200 running the `Qwen3.5-35B-A3B` Mixture-of-Experts backbone from 1,306 ms down to 65.12 ms—a 20.1x speedup that lifts theoretical control capacity from 4.62 Hz to 92.16 Hz. Under real-time evaluation, `Gamer-35B-A3B-CPT` actually improves by 3.4 percentage points over time-paused scoring rather than collapsing. ### How a Pixel-Only Agent Survives Long-Horizon RPG Quests Playing *Honkai: Star Rail* requires far more than reflex timing. Players must follow shifting quest objectives, talk to NPCs, track failed attempts, navigate multi-room zones, and clear logic or rhythm mini-games over long sessions that easily blow past a single context window. According to the [arXiv technical report](https://arxiv.org/abs/2611.18397) and its companion [Hugging Face Paper page](https://huggingface.co/papers/2611.18397), the 27-author team tackled this through a three-stage training pipeline built on `Qwen3-VL-8B` and `Qwen3.5-35B-A3B` backbones: - **Stage I — Continued Pre-Training (CPT):** The team trained on more than 10,000 hours of gameplay video paired with synchronized device-level keyboard and mouse logs. Two subtle training tricks prevent naive imitation collapse: a repeat-aware loss decay ($\gamma = 0.90$) that keeps long key-holds (like holding `W` to sprint) from drowning out crucial action transitions, and random key remapping per trajectory so the policy learns spatial intent rather than memorizing hardcoded bindings. - **Stage II — Sparse Reasoning and Cross-Context Memory (SFT):** Using roughly 1,000 hours of curated *Honkai: Star Rail* footage, the researchers added around 1.2 million "Think" annotations and 223,000 structured memory summaries across 63,000 activity segments. Crucially, Gamer does not burn tokens thinking on every frame; it triggers sparse reasoning only at route shifts, target switches, new prompts, or task transitions, while folding completed objectives and failed attempts into persistent memory summaries as older frames age out of context. - **Stage III — Online Reinforcement Learning (TA-GRPO):** Finally, Temporal-Aware Group Relative Policy Optimization refines memory-linked rollout segments against environment-verified rewards without requiring a separate value critic or learned reward model. On 40 held-out *Honkai: Star Rail* mini-game tasks spanning seven distinct puzzle types, `Gamer-RL` achieved a 49.7% macro-average success rate—solving tasks across all seven categories and improving by 10.5 percentage points over supervised fine-tuning alone (39.2%), while nearly tripling the strongest general-purpose baseline (`GPT-6.1-Sol`) at 17.1%. ### Automated QA vs. Live-Service Anti-Cheat Reality Open-sourcing an agent that plays a live-service gacha hit strictly through mouse and keyboard inputs naturally raises eyebrows among players worried about farming bots and automated account progression. Both the researchers and independent observers point to two constraints that separate Gamer from consumer cheat tools. First, the latency numbers that make real-time play viable were measured on datacenter NVIDIA H200 GPUs running MoE models, well beyond typical desktop hardware. Second, the team explicitly stated that it "does not endorse using Gamer for cheating, unauthorized access, exploiting security vulnerabilities, or any behavior that harms users or undermines fairness." For studios, the strategic upside sits elsewhere—a contrast to Western industry debates currently centered on generative asset pipelines (as recently examined in [Mushimu’s coverage of Ben Feder’s remarks](https://www.mushimu.com/post/9552)). By training an agent that perceives only rendered pixels and acts through standard OS inputs, miHoYo’s R&D division gains a general-purpose automated QA tester capable of verifying quest progression, catching geometry soft-locks, and benchmarking accessibility barriers across engine updates without writing bespoke test hooks for every game mode. Whether that pixel-only paradigm spreads from Chinese live-service powerhouses to broader studio QA pipelines will depend on how quickly inference costs fall from datacenter H200 clusters down to local workstation cards. ### Sources & Further Reading - [Automaton West (Japan)](https://automaton-media.com/en/news/mihoyo-develops-ai-model-that-can-play-complex-games-like-honkai-star-rail-just-like-a-human-would-with-mouse-and-keyboard-inputs/) - [arXiv Technical Report (2611.18397)](https://arxiv.org/abs/2611.18397) - [Gamer GitHub Repository](https://github.com/RainBowLuoCS/Gamer) - [Hugging Face Papers — Gamer (THUSI-Lab / Honkai AI R&D)](https://huggingface.co/papers/2611.18397) ~Gaming ~AI ~HonkaiStarRail ~miHoYo