Thursday 20 March 2025
Researchers have made significant strides in developing mobile AI agents that can interact with mobile devices and perform tasks autonomously. The latest breakthrough comes from a team of scientists who have created ReachAgent, a two-stage framework designed to improve task-completion abilities.
Mobile AI agents are increasingly being used to automate various tasks on smartphones and tablets, such as shopping, browsing, and communication. However, existing agents tend to focus on the most task-relevant elements at each step, leading to local optimal solutions and ignoring the overall GUI flow. ReachAgent addresses this issue by utilizing page reaching and page operation subtasks, along with reward-based preference GUI flows.
The framework consists of two stages: a first stage that uses a large-scale visual language model (VLM) as its backbone, followed by a second stage that incorporates reinforcement learning to fine-tune the agent’s task-completion abilities. The VLM is trained on a dataset called MobileReach, which breaks down tasks into page reaching and operation subtasks.
The researchers demonstrate the effectiveness of ReachAgent in several experiments, comparing it to four other baselines. The results show that ReachAgent significantly improves Intersection over Union (IoU) Accuracy and Text Accuracy by 7.12% and 7.69%, respectively, on step-level and 4.72% and 4.63% on task-level compared to the state-of-the-art agent.
One of the key innovations of ReachAgent is its ability to generalize across different GUI flows and tasks. This is achieved through a pre-defined template-based system that allows the agent to generate tasks based on user interactions. The researchers also developed a page reaching subtask generator, which splits the GUI flow into smaller subtasks and assigns tasks accordingly.
The potential applications of ReachAgent are vast, from automating routine tasks to assisting people with disabilities. For instance, an elderly person could use ReachAgent to place orders online or book appointments without having to navigate complex interfaces.
However, there are also limitations to the technology. The dataset used to train the VLM is limited to a specific set of GUI flows and tasks, which may not generalize well to other scenarios. Additionally, the reinforcement learning stage requires a large amount of data and computational resources, making it challenging to scale up for more complex tasks.
Despite these challenges, ReachAgent represents a significant step forward in the development of mobile AI agents.
Cite this article: “Introducing ReachAgent: A Two-Stage Framework for Improving Task-Completion Abilities in Mobile AI Agents”, The Science Archive, 2025.
Mobile Ai, Autonomous Agents, Task Completion, Gui Flows, Reinforcement Learning, Visual Language Model, Mobilereach Dataset, Intersection Over Union Accuracy, Text Accuracy, Template-Based System.
