+ Serena is under active development! See the latest updates, upcoming features, and lessons learned to stay up to date. +
+ + + + ### LLM Integration Serena provides the necessary [tools](#full-list-of-tools) for coding workflows, but an LLM is required to do the actual work, diff --git a/lessons_learned.md b/lessons_learned.md new file mode 100644 index 0000000..765b187 --- /dev/null +++ b/lessons_learned.md @@ -0,0 +1,61 @@ +# Lessons Learned + +In this document we briefly collect what we have learned while developing and using Serena, +what works well and what doesn't. + +## What Worked + +### Separate Tool Logic From MCP Implementation + +MCP is just another protocol, one should let the details of it creep into the application logic. +The official docs suggest using function annotations to define tools and prompts. While that may be +useful for small projects to get going fast, it is not wise for more serious projects. In Serena, +all tools are defined independently and then converted to instances of `MCPTool` using our `make_tool` +function. + +### Tempfiles and Snapshots for Testing of Editing Tools + +We test most aspects of Serena by having a small "project" for each supported language in `tests/resources`. +For the editing tools, which would change the code in these projects, we use tempfiles to copy over the code. +The pretty awesome [syrupy](https://github.com/syrupy-project/syrupy) pytest plugin helped in developing +snapshot tests. + +### Dashboard and GUI for Logging + +It is very useful to know what the MCP Server is doing. We collect and display logs in a GUI or a web dashboard, +which helps a lot in seeing what's going on and in identifying any issues. + +### Unrestricted Bash Tool + +We know it's not particularly safe to permit unlimited shell commands outside a sandbox, but we did quite some +evaluations and so far... nothing bad has happened. Seems like the current versions of the AI overlords rarely want to execute `sudo rm - rf /`. +Still, we are working on a safer approach as well as better integration with sandboxing. + +### Multilspy + +The [multilspy](https://github.com/microsoft/multilspy/) project helped us a lot in getting started and stands at the core of Serena. +Many more well known python implementations of language servers were subpar in code quality and design (for example, missing types). + +### Developing Serena with Serena + +We clearly notice that the better the tool gets, the easier it is to make it even better + +## What Didn't Work + +### Lifespan Handling by MCP Clients + +The MCP technology is clearly very green. Even though there is a lifespan context in the MCP SDK, +many clients, including Claude Desktop, fail to properly clean up, leaving zombie processes behind. +We mitigate this through the GUI window and the dashboard, so the user sees whether Serena is running +and can terminate it there. + +### Cross-OS Tkinter GUI + +Different OS have different limitations when it comes to starting a window or dealing with Tkinter +installations. This was so messy to get right that we pivoted to a web-dashboard instead + +### Editing Based on Line Numbers + +Not only are LLMs notoriously bad in counting, but also the line numbers change after edit operations, +and LLMs are also often too dumb to understand that they should update the line numbers information they had +received before. We pivoted to string-matching and symbol-name based editing. \ No newline at end of file diff --git a/roadmap.md b/roadmap.md new file mode 100644 index 0000000..4b936b3 --- /dev/null +++ b/roadmap.md @@ -0,0 +1,63 @@ +# Roadmap + +This document gives an overview of the ongoing and future development of Serena. +If you have a proposal or want to discuss something, feel free to open a discussion +on Github. For a summary of the past development, see the [changelog](/CHANGELOG.md). + +Want to see us reach our goals faster? You can help out with an issue, start a discussion, or +inform us about funding opportunities so that we can devote more time to the project. + +## Overall Goals + +Serena has the potential to be the go-to tool for most LLM coding tasks, since it is +unique in its ability to be used as MCP Server in any kind of environment +while still being a capable agent. We want to achieve the following goals in terms of functionality: + +1. Top performance (comparable to API-based coding agents) when used through official (free) clients like Claude Desktop. +1. Lowering API costs and potentially improving performance of coding clients (Claude Code, Codex, Cline, Roo, Cursor/Windsurf/VSCode etc). +1. Transparency and simplicity of use. Achieved through the dashboard/logging GUI. +1. Integrations with major frameworks that don't accept MCP. Usable as a library. + +Apart from the functional goals, we have the goal of having great code design, so that Serena can be viewed +as a reference for how to implement MCP Servers. Such projects are an emerging technology, and +best practices are yet to be determined. We will share our experiences in [lessons learned](/lessons_learned.md). + + +## Immediate/Ongoing + +- Support for projects using multiple programming languages. +- Evaluate whether `ReplaceLinesTool` can be removed in favor of a more reliable and performant editing approach. +- Generally experiment with various approaches to editing tools +- Manual evaluation on selected tasks from SWE-verified +- Manual evaluation of cost-lowering and performance when used within popular non-MCP agents +- Improvements in prompts, in particular giving examples and extending modes and contexts + +## Upcoming + +- Publishing Serena as a package that can also be used as library +- Use linting and type-hierarchy from the LSP in tools +- Tools for refactoring (rename, move) - speculative, maybe won't do this. +- Tracking edits and rolling them back with the dashboard +- Improve configurability and safety of shell tool. Maybe autogeneration of tools from a list of commands and descriptions. +- Transparent comparison with DesktopCommander and ... +- Automatic evaluation using OpenHands, submission to SWE-Bench +- Evaluation whether incorporating other MCPs increases performance or usability (memory bank is a candidate) +- More documentation and best practices + +## Stretch + +- Allow for sandboxing and parallel instances of Serena, maybe use openhands or codex for that +- Incorporate a verifier model or generally a second model (maybe for applying edits) as a tool. +- Building on the above, allow for the second model itself to be reachable through an MCP server, so it can be used for free +- Tracking edits performed with shell tools + +## Beyond Serena + +The technologies and approaches taken in Serena can be used for various research and service ideas. Some thought that we had are: + +- PR and issue assistant working with GitHub, similar to how [OpenHands](https://github.com/All-Hands-AI/OpenHands) + and [qodo](https://github.com/qodo-ai/pr-agent) operate. Should be callable through @serena +- Tuning a coding LLM with Serena's tools with RL on one-shot tasks. We would need compute-funding for that +- Develop a web app to quantitatively compare the performance of various agents by scraping PRs and manually crafted metadata. + The main metric for coding agents should be *developer experience*, and that is hard to grasp and is poorly correlated with + performance on current benchmarks. \ No newline at end of file