LaVague’s Open-Sourced Large Action Model Outperforms Gemini and ChatGPT in Information Retrieval: A Game Changer in AI Web Agents

LaVague's technology comprises several key components, including the World Model and Action Engine, which work together to automate tasks based on user-defined objectives.

1. World Model: The World Model is responsible for converting the user's global objective into the next instruction for the Action Engine to carry out based on visual and textual information1. It uses a multi-modal model to perform this conversion. By default, LaVague uses GPT-4o, but developers can use any LlamaIndex multi-modal LLM.

2. Action Engine: This component receives a natural language text instruction and generates the action needed to carry out the instruction. In the example shared in the web search context, the Action Engine performs Retrieval-Augmented Generation (RAG) and generates the code for the action using the default embedding and Large Language Models (OpenAI's text-embedding-3-large & GPT-4o).

3. Driver: The Driver component is used to perform actions on web browsers and get information about the current web page. LaVague currently provides a Selenium Driver component by default, as well as a Playwright Driver option.

4. Web Agent: The Web Agent brings all of these components together and can be used to perform tasks defined by the user's objective argument with the run method or to launch an interactive interface with Gradio with the demo() mode.

Here's a simplified workflow of how these components function together:

• The user provides a global objective.
• The World Model takes this objective and the current state of the webpage (provided by the Driver) to generate the next step, aka2. text instruction.
• This instruction is sent to the Action Engine, which then generates the automation code needed to perform this step and executes it.
• The World Model then receives new text and image data to reflect the updated state of the webpage. With this information, it is able to generate the next instruction needed to achieve the objective.
• This process repeats until the objective is achieved.

This architecture allows LaVague to automate tasks such as logging into an account and changing a username, as demonstrated in the example provided.

Developers can use LaVague to create an AI agent for retrieving the latest research papers on Hugging Face by following these steps:

1. Install LaVague: pip install lavague
2. Build a Web Agent:

from lavague.core import WorldModel, ActionEngine
from lavague.core.agents import WebAgent
from lavague.drivers.selenium import SeleniumDriver
selenium_driver = SeleniumDriver(headless=False)
world_model = WorldModel()
action_engine = ActionEngine(selenium_driver)
agent = WebAgent(world_model, action_engine)

3. Set the URL: agent.get("https://huggingface.co/docs")
4. Run the agent with the objective: agent.run("Go on the quicktour of PEFT")

LaVague's default OpenAI API configuration requires the OPENAI_API_KEY variable to be set in the local environment with a valid API key. This example demonstrates how LaVague can be used to create an AI agent that retrieves the latest research papers on Hugging Face.