ScreenEnv

ScreenEnv: Deploy Your Full Stack Desktop Agent

Published: July 10, 2025
Authors: Amir Mahla, Aymeric Roucher
Page Description: Aiming to advance and democratize AI through open source and scientific methods.

Overview of ScreenEnv

ScreenEnv is an innovative Python library designed to facilitate the creation of isolated desktop environments running on Linux, specifically Ubuntu, within Docker containers. This powerful tool allows users to develop, test, and deploy Graphical User Interface (GUI) agents—often referred to as Computer Use agents—with unprecedented ease. By providing a complete sandboxed desktop experience that can perform intricate automation tasks, ScreenEnv caters not only to developers but also to AI researchers looking to improve their applications with automation features.

Key Features of ScreenEnv

1. Full Desktop Control:

   • Automation of mouse and keyboard actions.
   • Management of window organization and application launching.
   • Capability to perform file operations, execute terminal commands, and record entire sessions.

2. Dual Integration Modes:

   • Supports the Model Context Protocol (MCP) for AI systems.
   • Offers a Direct Sandbox API for integrating with existing frameworks, allowing for flexibility based on user needs.

3. Docker Native Environment:

   • Eliminates the complexities associated with Virtual Machine (VM) setups.
   • Environment is fully isolated, easily reproducible, and deployable within seconds across various architecture including AMD64 and ARM64.

Simple Installation and Setup

To install ScreenEnv, users can simply execute:

pip install screenenv

Getting started with a basic setup involves just a single line of code:

from screenenv import Sandbox
sandbox = Sandbox()  # That's it!

Integration Approaches

ScreenEnv facilitates flexibility through two integration methodologies:

1. Direct Sandbox API

This option is optimal for users who prefer to retain maximum control while developing custom agent frameworks. A quick code snippet illustrates how:

from screenenv import Sandbox

sandbox = Sandbox(headless=False)
sandbox.launch("xfce4-terminal")
sandbox.write("echo 'Custom agent logic'")
screenshot = sandbox.screenshot()
...
sandbox.close()

2. MCP Server Integration

Designed specifically for AI systems utilizing the Model Context Protocol, this option allows for backend communication and control over the sandboxed desktop efficiently. The following code demonstrates how to start an MCP Server:

from screenenv import MCPRemoteServer
from mcp import ClientSession

server = MCPRemoteServer(headless=False)
print(f"MCP Server URL: {server.server_url}")

async def mcp_session():
    async with streamablehttp_client(server.server_url) as streams:
        async with ClientSession(*streams) as session:
            await session.initialize()
            response = await session.call_tool("screenshot", {})

Creating a Desktop Agent with ScreenEnv

ScreenEnv simplifies the process of automating desktop tasks by integrating seamlessly with "smolagents" to build custom desktop agents powered by different AI models.

Steps to Build Your Agent

1. Choose Your Model: Users can select from various AI backends, be it OpenAI’s API or Hugging Face models, to power their agents.

2. Define Your Custom Desktop Agent:

   • Extend the DesktopAgentBase class.
   • Implement the _setup_desktop_tools method to add specific functionalities (e.g., clicking, typing, opening applications).

from screenenv import DesktopAgentBase

class CustomDesktopAgent(DesktopAgentBase):
    ...
    def _setup_desktop_tools(self) -> None:
        ...

1. Run a Desktop Task:
   • Create an instance of your agent and invoke it as required.

sandbox = Sandbox(headless=False)
agent = CustomDesktopAgent(model=model, desktop=sandbox)
result = agent.run("Open a document and save.")
print(f"Result: {result}")

Who Can Benefit from ScreenEnv?

ScreenEnv appeals to a diverse range of users:

• AI Researchers and Developers: Those developing and testing AI applications that require user interaction with desktop systems.
• Automation Engineers: Professionals looking to optimize repetitive tasks with built agents that can manage desktop environments.
• Academic Institutions: Students and educators seeking a reproducible environment for learning and conducting experiments related to AI and automation.
• Software Testers: Individuals needing to automate GUI testing for applications without complex system setups.

Future Prospects

The ScreenEnv project is set to expand its capabilities beyond Linux systems, with plans to support platforms such as Android, macOS, and Windows. This future compatibility could provide developers and researchers with an unparalleled opportunity to create cross-platform GUI automation tools efficiently. The advancements will lead to highly reproducible, sandboxed environments ideal for evaluating and benchmarking various applications.

Conclusion

ScreenEnv is establishing itself as a crucial tool for those interested in automating desktop tasks with AI integration. By providing a straightforward setup, flexible integration methods, and the capacity for extensive desktop automation, it opens doors for innovation in a variety of fields. Whether you’re an AI researcher, a software developer, or someone keen to optimize desktop activities, ScreenEnv offers a robust and effective solution to enhance productivity and performance.

Explore more about ScreenEnv on GitHub and get involved in the community to leverage its rich feature set in your projects.