mirror of
https://github.com/tiennm99/serena.git
synced 2026-08-20 06:24:00 +00:00
Docs: roadmap, lessons learned. Buttons in readme
This commit is contained in:
Michael Panchenko
@@ -14,6 +14,23 @@ Note how Serena's tools enable Claude to find and edit the right symbols.
|
||||
|
||||
https://github.com/user-attachments/assets/6eaa9aa1-610d-4723-a2d6-bf1e487ba753
|
||||
|
||||
<p align="center">
|
||||
<em>Serena is under active development! See the latest updates, upcoming features, and lessons learned to stay up to date.</em>
|
||||
</p>
|
||||
<p align="center">
|
||||
<a href="CHANGELOG.md" style="text-decoration:none; display:inline-block; margin: 0 6px;">
|
||||
<span style="display:inline-block; padding:6px 14px; background:#1e293b; color:#fff; border-radius:5px; font-weight:bold; font-size:13px; text-decoration:none;">Changelog</span>
|
||||
</a>
|
||||
<a href="roadmap.md" style="text-decoration:none; display:inline-block; margin: 0 6px;">
|
||||
<span style="display:inline-block; padding:6px 14px; background:#14532d; color:#fff; border-radius:5px; font-weight:bold; font-size:13px; text-decoration:none;">Roadmap</span>
|
||||
</a>
|
||||
<a href="lessons_learned.md" style="text-decoration:none; display:inline-block; margin: 0 6px;">
|
||||
<span style="display:inline-block; padding:6px 14px; background:#7c4700; color:#fff; border-radius:5px; font-weight:bold; font-size:13px; text-decoration:none;">Lessons Learned</span>
|
||||
</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
### LLM Integration
|
||||
|
||||
Serena provides the necessary [tools](#full-list-of-tools) for coding workflows, but an LLM is required to do the actual work,
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Lessons Learned
|
||||
|
||||
In this document we briefly collect what we have learned while developing and using Serena,
|
||||
what works well and what doesn't.
|
||||
|
||||
## What Worked
|
||||
|
||||
### Separate Tool Logic From MCP Implementation
|
||||
|
||||
MCP is just another protocol, one should let the details of it creep into the application logic.
|
||||
The official docs suggest using function annotations to define tools and prompts. While that may be
|
||||
useful for small projects to get going fast, it is not wise for more serious projects. In Serena,
|
||||
all tools are defined independently and then converted to instances of `MCPTool` using our `make_tool`
|
||||
function.
|
||||
|
||||
### Tempfiles and Snapshots for Testing of Editing Tools
|
||||
|
||||
We test most aspects of Serena by having a small "project" for each supported language in `tests/resources`.
|
||||
For the editing tools, which would change the code in these projects, we use tempfiles to copy over the code.
|
||||
The pretty awesome [syrupy](https://github.com/syrupy-project/syrupy) pytest plugin helped in developing
|
||||
snapshot tests.
|
||||
|
||||
### Dashboard and GUI for Logging
|
||||
|
||||
It is very useful to know what the MCP Server is doing. We collect and display logs in a GUI or a web dashboard,
|
||||
which helps a lot in seeing what's going on and in identifying any issues.
|
||||
|
||||
### Unrestricted Bash Tool
|
||||
|
||||
We know it's not particularly safe to permit unlimited shell commands outside a sandbox, but we did quite some
|
||||
evaluations and so far... nothing bad has happened. Seems like the current versions of the AI overlords rarely want to execute `sudo rm - rf /`.
|
||||
Still, we are working on a safer approach as well as better integration with sandboxing.
|
||||
|
||||
### Multilspy
|
||||
|
||||
The [multilspy](https://github.com/microsoft/multilspy/) project helped us a lot in getting started and stands at the core of Serena.
|
||||
Many more well known python implementations of language servers were subpar in code quality and design (for example, missing types).
|
||||
|
||||
### Developing Serena with Serena
|
||||
|
||||
We clearly notice that the better the tool gets, the easier it is to make it even better
|
||||
|
||||
## What Didn't Work
|
||||
|
||||
### Lifespan Handling by MCP Clients
|
||||
|
||||
The MCP technology is clearly very green. Even though there is a lifespan context in the MCP SDK,
|
||||
many clients, including Claude Desktop, fail to properly clean up, leaving zombie processes behind.
|
||||
We mitigate this through the GUI window and the dashboard, so the user sees whether Serena is running
|
||||
and can terminate it there.
|
||||
|
||||
### Cross-OS Tkinter GUI
|
||||
|
||||
Different OS have different limitations when it comes to starting a window or dealing with Tkinter
|
||||
installations. This was so messy to get right that we pivoted to a web-dashboard instead
|
||||
|
||||
### Editing Based on Line Numbers
|
||||
|
||||
Not only are LLMs notoriously bad in counting, but also the line numbers change after edit operations,
|
||||
and LLMs are also often too dumb to understand that they should update the line numbers information they had
|
||||
received before. We pivoted to string-matching and symbol-name based editing.
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
# Roadmap
|
||||
|
||||
This document gives an overview of the ongoing and future development of Serena.
|
||||
If you have a proposal or want to discuss something, feel free to open a discussion
|
||||
on Github. For a summary of the past development, see the [changelog](/CHANGELOG.md).
|
||||
|
||||
Want to see us reach our goals faster? You can help out with an issue, start a discussion, or
|
||||
inform us about funding opportunities so that we can devote more time to the project.
|
||||
|
||||
## Overall Goals
|
||||
|
||||
Serena has the potential to be the go-to tool for most LLM coding tasks, since it is
|
||||
unique in its ability to be used as MCP Server in any kind of environment
|
||||
while still being a capable agent. We want to achieve the following goals in terms of functionality:
|
||||
|
||||
1. Top performance (comparable to API-based coding agents) when used through official (free) clients like Claude Desktop.
|
||||
1. Lowering API costs and potentially improving performance of coding clients (Claude Code, Codex, Cline, Roo, Cursor/Windsurf/VSCode etc).
|
||||
1. Transparency and simplicity of use. Achieved through the dashboard/logging GUI.
|
||||
1. Integrations with major frameworks that don't accept MCP. Usable as a library.
|
||||
|
||||
Apart from the functional goals, we have the goal of having great code design, so that Serena can be viewed
|
||||
as a reference for how to implement MCP Servers. Such projects are an emerging technology, and
|
||||
best practices are yet to be determined. We will share our experiences in [lessons learned](/lessons_learned.md).
|
||||
|
||||
|
||||
## Immediate/Ongoing
|
||||
|
||||
- Support for projects using multiple programming languages.
|
||||
- Evaluate whether `ReplaceLinesTool` can be removed in favor of a more reliable and performant editing approach.
|
||||
- Generally experiment with various approaches to editing tools
|
||||
- Manual evaluation on selected tasks from SWE-verified
|
||||
- Manual evaluation of cost-lowering and performance when used within popular non-MCP agents
|
||||
- Improvements in prompts, in particular giving examples and extending modes and contexts
|
||||
|
||||
## Upcoming
|
||||
|
||||
- Publishing Serena as a package that can also be used as library
|
||||
- Use linting and type-hierarchy from the LSP in tools
|
||||
- Tools for refactoring (rename, move) - speculative, maybe won't do this.
|
||||
- Tracking edits and rolling them back with the dashboard
|
||||
- Improve configurability and safety of shell tool. Maybe autogeneration of tools from a list of commands and descriptions.
|
||||
- Transparent comparison with DesktopCommander and ...
|
||||
- Automatic evaluation using OpenHands, submission to SWE-Bench
|
||||
- Evaluation whether incorporating other MCPs increases performance or usability (memory bank is a candidate)
|
||||
- More documentation and best practices
|
||||
|
||||
## Stretch
|
||||
|
||||
- Allow for sandboxing and parallel instances of Serena, maybe use openhands or codex for that
|
||||
- Incorporate a verifier model or generally a second model (maybe for applying edits) as a tool.
|
||||
- Building on the above, allow for the second model itself to be reachable through an MCP server, so it can be used for free
|
||||
- Tracking edits performed with shell tools
|
||||
|
||||
## Beyond Serena
|
||||
|
||||
The technologies and approaches taken in Serena can be used for various research and service ideas. Some thought that we had are:
|
||||
|
||||
- PR and issue assistant working with GitHub, similar to how [OpenHands](https://github.com/All-Hands-AI/OpenHands)
|
||||
and [qodo](https://github.com/qodo-ai/pr-agent) operate. Should be callable through @serena
|
||||
- Tuning a coding LLM with Serena's tools with RL on one-shot tasks. We would need compute-funding for that
|
||||
- Develop a web app to quantitatively compare the performance of various agents by scraping PRs and manually crafted metadata.
|
||||
The main metric for coding agents should be *developer experience*, and that is hard to grasp and is poorly correlated with
|
||||
performance on current benchmarks.
|
||||
Reference in New Issue
Block a user