Meet Tarsier: An Open Source Python Library to Enable Web Interaction with Multi-Modal LLMs like GPT4

As AI continues to grow and impact all aspects of our lives, research is being conducted to make it more useful and convenient. Today, AI is finding its utility in all dimensions of daily life. Extensive research has been conducted in varied fields. Consequently, the researchers of Reworkd have formulated Tarsier, an open-source Python library to facilitate web interaction with multi-modal Language Models (LLMs) like GPT-4.

Tarsier acts as a bridge, which enhances the capabilities of these models by visually tagging interactable elements on a web page and enabling interaction between users and machines.

Tarsier simplifies the intricate process of web interaction for LLMs. It is achieved by visually tagging elements using brackets and unique identifiers, such as IDs. These elements, encompassing buttons, links, and input fields visible on the page, establish a crucial mapping for GPT-4 to perform actions. In other words, Tarsier serves as a translator, making the web comprehensible to language models.

One feature of Tarsier is its ability to represent the page visually. This feature becomes important as existing vision language models face challenges. By offering Optical Character Recognition (OCR) utilities, Tarsier converts a page screenshot into a whitespace-structured string, ensuring that even non-multi-modal LLMs can grasp the content and meaning of a web page.

Tarsier introduces two fundamental utilities that significantly enhance the interaction capabilities of language models. These are Tagging Interactable Elements and Parsing Screenshots into OCR Text Representation.

Tarsier stands out in its capacity to tag interactable elements with a unique identifier. This identifier enables Language Models (LLMs) to understand the elements they can work with, like clicking buttons, following links, or completing input fields. This tagging method improves comprehension and creates a clear link from the LLM’s choices to the underlying elements on the web page.

Another revolutionary feature of Tarsier is its ability to convert screenshots into a spatially aware OCR text representation. This advancement allows the utilization of models like GPT-4 or any text-only LLM for web tasks, even if visual capabilities are absent. Essentially, Tarsier broadens the horizons of AI applications by enabling language models to engage with the web without relying on vision.

Also, Tarsier has a set of cookbooks that show how to use it with well-known LLM libraries like Langchain and LlamaIndex, making the onboarding process easier. These cookbooks let people experience Tarsier’s features directly by offering useful examples and insights.

In conclusion, Tarsier is a necessary tool to advance the capabilities of LLMs. It gives LLMs the tools to explore and comprehend the complexities of the web by offering an organized depiction of online elements. With its OCR tools, this capability is further extended to text-only models, removing obstacles and promoting a more diverse and adaptable AI environment.