There’s a lot of great stuff in here we think! We cite over 100 papers and websites. One thing I am very happy about is how easy it is to follow links in our survey to the bibliography which then links to the papers directly.

Then we talk about the LLM-Agent approaches and try to explain and make some sense of the many components that make up an LLM-based Computer Use Agent.

We then spend a lot of time looking at the different earlier (Pre-LLM) approaches to the problem, including the RL-from scratch period and even the very earliest planning-based approaches.

We try to categorize all the environments and datasets in common use and let users click/filter and browse through each of the datasets.

First, we try to ground our survey, say what we even mean by “Computer Use” and define some key terms, grounded in the classical agent-environment framework.

You can view the survey here: kennethmarino.com/computeruse/...
We tried to make it as interactive and fun as possible, including a retro DOS theme to go along with the subject.
Credit to Claude for helping me create the website :)

Super excited that the Computer Use survey I've been working on w/ @anamarasovic.bsky.social for a while now is ready! Originally we were planning on a more traditional survey paper but as more surveys came out we decided on an interactive website survey.

We’ll be at our poster soon if you’re interested in multimodal agents and continual learning.

Can’t take a lot of credit, was only an advisor on the project. Gabriel Sarch (PhD candidate at CMU) did a great job on this project; you should come talk to him!

Link to paper: arxiv.org/pdf/2406.14596

Interesting. I reposted my Bluesky handle again on the bird app without the links and keywords and, huh, it already has triple the engagement in less than an hour.
Wonder what could be going on 🤔🤔🤔