• bitcoinBitcoin(BTC)$58,961.00-5.18%
  • ethereumEthereum(ETH)$1,617.04-8.17%
  • tetherTether(USDT)$1.00-0.07%
  • binancecoinBNB(BNB)$538.93-7.14%
  • usd-coinUSDC(USDC)$1.00-0.03%
  • rippleXRP(XRP)$0.91-5.12%
  • solanaSolana(SOL)$68.99-5.63%
  • tronTRON(TRX)$0.301876-2.72%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.02-0.80%
  • HyperliquidHyperliquid(HYPE)$61.25-8.62%
  • zcashZcash(ZEC)$748.46-15.92%
  • moneroMonero(XMR)$454.73-8.13%
  • dogecoinDogecoin(DOGE)$0.054287-6.69%
  • leo-tokenLEO Token(LEO)$9.15-1.22%
  • whitebitWhiteBIT Coin(WBT)$56.40-6.47%
  • chainlinkChainlink(LINK)$8.55-4.84%
  • USDSUSDS(USDS)$1.000.03%
  • cardanoCardano(ADA)$0.142959-5.64%
  • Ethena USDeEthena USDe(USDE)$1.00-0.06%
  • stellarStellar(XLM)$0.128542-2.88%
  • daiDai(DAI)$1.000.02%
  • bitcoin-cashBitcoin Cash(BCH)$194.13-5.82%
  • nearNEAR Protocol(NEAR)$2.87-10.16%
  • litecoinLitecoin(LTC)$40.52-4.08%
  • CantonCanton(CC)$0.075163-6.49%
  • avalanche-2Avalanche(AVAX)$6.56-5.98%
  • USD1USD1(USD1)$1.00-0.06%
  • Ripple USDRipple USD(RLUSD)$1.000.02%
  • tether-goldTether Gold(XAUT)$3,528.63-2.68%
  • the-open-networkGram (prev. Toncoin)(GRAM)$0.98-3.79%
  • suiSui(SUI)$0.65-7.76%
  • uniswapUniswap(UNI)$4.43-10.37%
  • RainRain(RAIN)$0.003715-15.61%
  • Global DollarGlobal Dollar(USDG)$1.000.01%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.150.08%
  • paypal-usdPayPal USD(PYUSD)$1.00-0.01%
  • hedera-hashgraphHedera(HBAR)$0.051595-4.75%
  • shiba-inuShiba Inu(SHIB)$0.000004-5.20%
  • BittensorBittensor(TAO)$180.05-6.16%
  • crypto-com-chainCronos(CRO)$0.039866-3.41%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.120.06%
  • quant-networkQuant(QNT)$138.76-7.44%
  • Circle USYCCircle USYC(USYC)$1.140.05%
  • MemeCoreMemeCore(M)$0.863.33%
  • Falcon USDFalcon USD(USDF)$1.00-0.07%
  • okbOKB(OKB)$79.05-8.32%
  • United StablesUnited Stables(U)$1.00-0.01%
  • pax-goldPAX Gold(PAXG)$3,529.96-2.60%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
  • aaveAave(AAVE)$97.20-7.20%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

Alibaba Researchers Introduce Mobile-Agent: An Autonomous Multi-Modal Mobile Device Agent

February 4, 2024
in AI & Technology
Reading Time: 5 mins read
A A
Alibaba Researchers Introduce Mobile-Agent: An Autonomous Multi-Modal Mobile Device Agent
ShareShareShareShareShare

Mobile device agents utilizing Multimodal Large Language Models (MLLM) have gained popularity due to the rapid advancements in MLLMs, showcasing notable visual comprehension capabilities. This progress has made MLLM-based agents viable for diverse applications. The emergence of mobile device agents represents a novel application, requiring these agents to operate devices based on screen content and user instructions.

Existing work highlights the capabilities of Large Language Model (LLM)-based agents in task planning. However, challenges persist, particularly in the mobile device agent domain. While MLLMs show promise, including GPT-4V, they lack sufficient visual perception for effective mobile device operations. Previous attempts utilized interface layout files for localization but faced limitations in file accessibility, hindering their effectiveness.

Beijing Jiaotong University and Alibaba Group researchers have introduced Mobile-Agent, an autonomous multi-modal mobile device agent. Their approach utilizes visual perception tools to accurately identify and locate visual and textual elements within an app’s front-end interface. Leveraging the perceived vision context, Mobile-Agent autonomously plans and decomposes complex operation tasks, navigating through mobile apps step by step. Mobile-Agent differs from previous solutions by eliminating reliance on XML files or mobile system metadata, offering enhanced adaptability across diverse mobile operating environments through a vision-centric approach.

Mobile-Agent employs OCR tools for text and CLIP for icon localization. The framework defines eight operations, enabling the agent to perform tasks such as opening apps, clicking text or icons, typing, and navigating. The Mobile Agent exhibits iterative self-planning and self-reflection, enhancing task completion through user instructions and real-time screen analysis. The mobile agent completes each step of the operation iteratively. Before the iteration begins, the user needs to input an instruction. During the iteration, the agent may encounter errors, leading to the inability to complete the instruction. To improve the success rate of instruction, there is a self-reflection method.

The researchers presented Mobile-Eval, a benchmark of 10 popular mobile apps with three instructions each to evaluate Mobile-Agent comprehensively. The framework achieved completion rates of 91%, 82%, and 82% across instructions, with a high Process Score of around 80%. Relative Efficiency demonstrated Mobile-Agent’s 80% capability compared to human-operated steps. The results highlight the effectiveness of Mobile-Agent, showcasing its self-reflective capabilities in correcting errors during the execution of instructions, contributing to its robust performance as a mobile device assistant.

To sum up, Beijing Jiaotong University and Alibaba Group researchers have introduced Mobile-Agent, an autonomous multimodal agent proficient in operating diverse mobile applications through a unified visual perception framework. By precisely identifying and locating visual and textual elements within app interfaces, Mobile-Agent autonomously plans and executes tasks. Its vision-centric approach enhances adaptability across mobile operating environments, eliminating the need for system-specific customizations. The study demonstrates Mobile-Agent’s effectiveness and efficiency through experiments, highlighting its potential as a versatile and adaptable solution for language-agnostic interaction with mobile applications.


Check out the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our Telegram Channel


YOU MAY ALSO LIKE

CrowdStrike Reportedly in Talks to Acquire AI Security Partner Zscaler – Unite.AI

How To Stream GTA 6 On Your iPhone, iPad Or Mac

Asjad is an intern consultant at Marktechpost. He is persuing B.Tech in mechanical engineering at the Indian Institute of Technology, Kharagpur. Asjad is a Machine learning and deep learning enthusiast who is always researching the applications of machine learning in healthcare.


🎯 [FREE AI WEBINAR] ‘Using ANN for Vector Search at Speed & Scale (Demo on AWS)’ (Feb 5, 2024)


Credit: Source link

ShareTweetSendSharePin

Related Posts

CrowdStrike Reportedly in Talks to Acquire AI Security Partner Zscaler – Unite.AI
AI & Technology

CrowdStrike Reportedly in Talks to Acquire AI Security Partner Zscaler – Unite.AI

November 20, 2026
How To Stream GTA 6 On Your iPhone, iPad Or Mac
AI & Technology

How To Stream GTA 6 On Your iPhone, iPad Or Mac

November 20, 2026
This GTA 6 Fast Travel Trick Comes At A Price
AI & Technology

This GTA 6 Fast Travel Trick Comes At A Price

November 20, 2026
RadixArk Ships DaisyRose-27B, a 27B Agentic Coding Model Trained With Reinforcement Learning Only
AI & Technology

RadixArk Ships DaisyRose-27B, a 27B Agentic Coding Model Trained With Reinforcement Learning Only

November 20, 2026
Next Post
First Interstate BancSystem: 2024 Does Not Bode Well (NASDAQ:FIBK)

First Interstate BancSystem: 2024 Does Not Bode Well (NASDAQ:FIBK)

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
NYC passes bill to ban horse-drawn carriages 

NYC passes bill to ban horse-drawn carriages 

November 18, 2026
Mineros S.A. 2026 Q3 – Results – Earnings Call Presentation (OTCMKTS:MNSAF) 2026-11-17

Mineros S.A. 2026 Q3 – Results – Earnings Call Presentation (OTCMKTS:MNSAF) 2026-11-17

November 17, 2026
AI startup Anthropic unveils its public IPO filing for the first time

AI startup Anthropic unveils its public IPO filing for the first time

November 16, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!