mirror of
https://github.com/tiennm99/serena.git
synced 2026-08-16 14:23:22 +00:00
Memories [ci skip]
This commit is contained in:
Michael Panchenko
@@ -0,0 +1,123 @@
|
||||
# Current Architecture: Solid-LSP Implementation
|
||||
|
||||
## Overview
|
||||
|
||||
Solid-LSP (`src/solidlsp/`) is Serena's current language server integration layer, designed as a **simplified, deadlock-resistant** replacement for the problematic multilspy architecture. It enables **single-process operation** with MCP servers while maintaining all semantic code analysis capabilities.
|
||||
|
||||
## Current System Status
|
||||
|
||||
- **Default Implementation**: Solid-LSP is the only language server implementation (multilspy completely removed)
|
||||
- **Process Isolation**: **Disabled by default** (`USE_PROCESS_ISOLATION = False`)
|
||||
- **MCP Integration**: Runs safely in the same process as MCP server
|
||||
- **Performance**: Lower latency and resource usage than process-isolated multilspy
|
||||
|
||||
## Core Architecture
|
||||
|
||||
### Primary Components
|
||||
|
||||
#### 1. **`SolidLanguageServer`** (`src/solidlsp/ls.py`)
|
||||
- **1,634 lines**: Main language server interface
|
||||
- **Enhanced Methods**: Same semantic capabilities as multilspy but with improved async handling
|
||||
- **Lifecycle Control**: `start()`, `stop()`, `is_running()`, `language_server()` property
|
||||
- **Clean Async Patterns**: No coroutine leakage between async contexts
|
||||
|
||||
#### 2. **`SolidLanguageServerHandler`** (`src/solidlsp/ls_handler.py`)
|
||||
- **512 lines**: Simplified LSP protocol handler
|
||||
- **Direct Process Management**: More straightforward than multilspy's complex orchestration
|
||||
- **Resource Management**: Better cleanup and process termination
|
||||
|
||||
#### 3. **Protocol Layer** (`src/solidlsp/lsp_protocol_handler/`)
|
||||
- **Simplified Protocol**: Direct LSP communication without excessive abstraction
|
||||
- **Types and Constants**: LSP type definitions and protocol constants
|
||||
- **Request Handling**: Streamlined request/response pattern
|
||||
|
||||
### Language Server Support
|
||||
|
||||
Solid-LSP maintains **identical language support** to multilspy:
|
||||
|
||||
#### Primary Languages (Directly Supported)
|
||||
- **Python**: Pyright Language Server (`src/solidlsp/language_servers/pyright_language_server/`)
|
||||
- **Java**: Eclipse JDTLS (`src/solidlsp/language_servers/eclipse_jdtls/`)
|
||||
- **TypeScript/JavaScript**: TypeScript Language Server (`src/solidlsp/language_servers/typescript_language_server/`)
|
||||
|
||||
#### Additional Languages (Full Support)
|
||||
- **C#**: OmniSharp (`src/solidlsp/language_servers/omnisharp/`)
|
||||
- **Rust**: Rust-Analyzer (`src/solidlsp/language_servers/rust_analyzer/`)
|
||||
- **Go**: Gopls (`src/solidlsp/language_servers/gopls/`)
|
||||
- **Ruby**: Solargraph (`src/solidlsp/language_servers/solargraph/`)
|
||||
- **C++**: Clangd (`src/solidlsp/language_servers/clangd_language_server/`)
|
||||
- **Dart**: Dart Language Server (`src/solidlsp/language_servers/dart_language_server/`)
|
||||
- **PHP**: Intelephense (`src/solidlsp/language_servers/intelephense/`)
|
||||
- **Kotlin**: Kotlin Language Server (`src/solidlsp/language_servers/kotlin_language_server/`)
|
||||
|
||||
## Key Architectural Improvements
|
||||
|
||||
### 1. **Simplified Async Patterns**
|
||||
- **Clean Boundaries**: Proper async context management prevents MCP server contamination
|
||||
- **No Coroutine Leakage**: Eliminates the unawaited coroutine warnings from multilspy
|
||||
- **Direct Operations**: Fewer abstraction layers where async context could be corrupted
|
||||
|
||||
### 2. **Single Process Safety**
|
||||
- **Safe MCP Integration**: Designed to work within MCP server process without deadlocks
|
||||
- **Eliminated IPC Overhead**: Direct method calls instead of inter-process communication
|
||||
- **Resource Efficiency**: Lower memory usage and faster startup times
|
||||
|
||||
### 3. **Enhanced Process Control**
|
||||
- **Direct Process Management**: `_start_server_process()`, `_start_server()` methods
|
||||
- **Better State Management**: `_server_context` attribute for enhanced control
|
||||
- **Improved Cleanup**: More reliable resource management and process termination
|
||||
|
||||
## Integration Points
|
||||
|
||||
### MCP Server Integration
|
||||
Current implementation in `src/serena/mcp.py:590`:
|
||||
```python
|
||||
if not USE_PROCESS_ISOLATION:
|
||||
mcp_factory = SerenaMCPFactorySingleProcess(context=context, project=project_file)
|
||||
else:
|
||||
mcp_factory = SerenaMCPFactoryWithProcessIsolation(context=context, project=project_file)
|
||||
```
|
||||
|
||||
### Agent Integration
|
||||
Language server creation in `src/serena/agent.py:642-689`:
|
||||
```python
|
||||
def create_ls_for_project(...) -> SolidLanguageServer:
|
||||
# Creates LanguageServerConfig with project settings
|
||||
# Returns SolidLanguageServer.create() instance
|
||||
```
|
||||
|
||||
## Configuration and Settings
|
||||
|
||||
### Core Configuration
|
||||
- **`USE_PROCESS_ISOLATION = False`**: Process isolation disabled by default
|
||||
- **Language Detection**: Automatic based on project composition
|
||||
- **Timeout Settings**: Configurable language server timeouts
|
||||
- **LSP Communication Tracing**: Optional debugging support
|
||||
|
||||
### Project-Level Settings
|
||||
- **Ignored Paths**: Respects project configuration and gitignore
|
||||
- **Language Selection**: Automatic or manual language server selection
|
||||
- **Timeout Configuration**: Per-project timeout settings
|
||||
|
||||
## Performance Benefits
|
||||
|
||||
### vs. Multilspy + Process Isolation
|
||||
1. **Lower Latency**: Direct method calls instead of IPC
|
||||
2. **Reduced Memory**: Single process instead of multiple processes
|
||||
3. **Faster Startup**: No process creation overhead
|
||||
4. **Simpler Debugging**: All components in same process with unified stack traces
|
||||
|
||||
### Stability Improvements
|
||||
1. **No Asyncio Deadlocks**: Clean async boundaries prevent contamination
|
||||
2. **Reliable Operations**: `find_symbol` and other tools work consistently
|
||||
3. **Resource Management**: Better cleanup prevents resource leaks
|
||||
4. **Error Handling**: Simplified error propagation and handling
|
||||
|
||||
## Current Operational State
|
||||
|
||||
- **Production Ready**: Default implementation for all Serena deployments
|
||||
- **Fully Tested**: All semantic tools (`find_symbol`, `replace_symbol_body`, etc.) working reliably
|
||||
- **MCP Compatible**: Stable operation with Claude Desktop and other MCP clients
|
||||
- **Cross-Platform**: Works on all supported operating systems
|
||||
|
||||
This architecture represents the successful resolution of the multilspy asyncio contamination issues while maintaining full semantic code analysis capabilities and improving overall performance.
|
||||
@@ -1,63 +0,0 @@
|
||||
# Deadlock Analysis: MCP Symbolic Tools Timeout Issue
|
||||
|
||||
## Problem Summary
|
||||
Intermittent deadlocks occur when using symbolic tools (particularly `find_symbol`) through MCP clients. Successful calls are followed by calls that hang for 240 seconds before timing out. Once a timeout occurs, the language server calls will no longer respond.
|
||||
|
||||
## ROOT CAUSE DISCOVERED: Asyncio Event Loop Interference
|
||||
|
||||
### Critical Evidence from MCP Server Logs
|
||||
The smoking gun was found in the MCP server logs showing **unawaited coroutines**:
|
||||
|
||||
```
|
||||
INFO mcp.server.lowlevel.server:_handle_message:524 - Warning: RuntimeWarning: coroutine 'LanguageServer.request_full_symbol_tree' was never awaited
|
||||
```
|
||||
|
||||
This warning is being logged by the **MCP server's `_handle_message` method**, not SerenaAgent code, indicating coroutines are being created in the MCP server's asyncio context but never properly awaited.
|
||||
|
||||
### The Real Issue: Dual Asyncio Context Contamination
|
||||
|
||||
**Architecture Problem:**
|
||||
1. **MCP Server**: Runs its own asyncio event loop to handle incoming requests
|
||||
2. **SerenaAgent**: Creates its own asyncio event loop in separate thread for language serve, and even a second loop for the dashboard if so configured
|
||||
3. **Conflict**: When MCP server calls SerenaAgent tools, we have **two or three asyncio contexts interacting**
|
||||
|
||||
**Deadlock Mechanism:**
|
||||
```
|
||||
MCP Server (asyncio loop A)
|
||||
→ handles tool request
|
||||
→ calls SerenaAgent.find_symbol()
|
||||
→ SerenaAgent uses asyncio.run_coroutine_threadsafe()
|
||||
→ Creates coroutine in language server loop (loop B)
|
||||
→ BUT: Coroutine gets leaked into MCP's context and never awaited
|
||||
→ Dangling coroutines accumulate in MCP server's loop
|
||||
→ Eventually causes resource exhaustion/event loop blocking
|
||||
→ Language server appears to "hang" but it's actually MCP loop contamination
|
||||
```
|
||||
|
||||
### Discarded Hypotheses
|
||||
- **Not a timeout issue**: Language server actually works fine in isolation
|
||||
- **Not abandoned threads**: The threading mechanism works correctly
|
||||
- **Not LSP deadlock**: The TypeScript language server itself isn't hanging
|
||||
|
||||
The actual issue is **asyncio context bleeding** between MCP server and SerenaAgent.
|
||||
|
||||
### Why We Can't Reproduce Outside MCP
|
||||
- **Direct script calls**: No MCP server, no dual asyncio context
|
||||
- **Single event loop**: Only SerenaAgent's language server loop exists
|
||||
- **No async interference**: Clean, isolated execution
|
||||
- **MCP-specific**: Requires the exact async context interaction pattern
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Key Files and Locations
|
||||
- `src/multilspy/language_server.py:1870` - `run_coroutine_threadsafe` creates coroutines that leak to MCP context
|
||||
- MCP server `_handle_message` - Where unawaited coroutine warnings appear
|
||||
|
||||
## Solution
|
||||
**Process Isolation** is the fundamental fix:
|
||||
1. **Separate processes**: MCP server, SerenaAgent and Dashboard in different processes
|
||||
2. **IPC communication**: Replace direct method calls with inter-process communication
|
||||
3. **Clean async boundaries**: Each process manages its own asyncio context
|
||||
4. **No coroutine leakage**: Complete isolation prevents context contamination
|
||||
|
||||
This explains why the deadlock is **MCP-specific** and doesn't occur in direct tool execution - it's fundamentally about asyncio context contamination between the MCP server and SerenaAgent.
|
||||
@@ -1,16 +0,0 @@
|
||||
# Available Language Servers
|
||||
|
||||
Serena supports multiple programming languages through language servers. The language servers currently available in the project are:
|
||||
|
||||
## Directly Supported (Out of the Box)
|
||||
- **Python**: Using Pyright Language Server (previously used Jedi)
|
||||
- **Java**: Using Eclipse JDTLS (Note: startup is slow, especially initial startup)
|
||||
- **TypeScript/JavaScript**: Using TypeScript Language Server
|
||||
|
||||
## Indirectly Supported (May Require Manual Setup)
|
||||
- **Ruby**: Using Solargraph
|
||||
- **Go**: Using Gopls
|
||||
- **C#**: Using OmniSharp
|
||||
- **Rust**: Using Rust-Analyzer
|
||||
|
||||
These language servers utilize the Language Server Protocol (LSP) to provide semantic code analysis capabilities. The system can be extended to support additional languages by providing adapters for new language server implementations.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Historical Context: Multilspy Era and Migration to Solid-LSP
|
||||
|
||||
## What multilspy Was
|
||||
|
||||
Multilspy was Serena's original language server integration layer that provided semantic code analysis through Language Server Protocol (LSP). It was located in `src/multilspy/` and included:
|
||||
|
||||
### Core Components (No Longer Present)
|
||||
- **`multilspy.LanguageServer`**: Main language server interface with async methods like `request_full_symbol_tree()`
|
||||
- **`LanguageServerHandler`**: Complex async task management and protocol handling
|
||||
- **`lsp_protocol_handler`**: Separate protocol abstraction layer
|
||||
- **Process Management**: Complex threading and async orchestration
|
||||
|
||||
### Supported Languages
|
||||
Multilspy supported the same languages now supported by solid-lsp:
|
||||
- Python (Pyright), Java (Eclipse JDTLS), TypeScript/JavaScript, C#, Rust, Go, Ruby, C++, Dart, PHP, Kotlin
|
||||
|
||||
## Why We Moved Away from Multilspy
|
||||
|
||||
### 1. **Asyncio Contamination Issues**
|
||||
- **Root Problem**: MCP server runs its own asyncio event loop, while multilspy created additional asyncio loops
|
||||
- **Coroutine Leakage**: `multilspy.LanguageServer.request_full_symbol_tree()` coroutines leaked into MCP server context but were never awaited
|
||||
- **Evidence**: MCP server logs showed: `RuntimeWarning: coroutine 'LanguageServer.request_full_symbol_tree' was never awaited`
|
||||
- **Result**: Accumulated unawaited coroutines caused resource exhaustion and apparent "hangs"
|
||||
|
||||
### 2. **Dual Event Loop Interference**
|
||||
- **MCP Server**: Asyncio loop A for handling requests
|
||||
- **SerenaAgent**: Asyncio loop B for language servers + loop C for dashboard
|
||||
- **Conflict**: Multiple asyncio contexts caused contamination and deadlocks
|
||||
|
||||
### 3. **Complex Architecture**
|
||||
- **Over-abstraction**: Multiple layers of protocol handling increased complexity
|
||||
- **Threading Issues**: Complex async task management made debugging difficult
|
||||
- **Resource Overhead**: Separate protocol handler processes added overhead
|
||||
|
||||
### 4. **MCP-Specific Problems**
|
||||
- **Only in MCP environments**: Issues didn't occur in direct script execution
|
||||
- **Mandatory Process Isolation**: Required separate processes to prevent asyncio contamination
|
||||
- **Performance Penalty**: IPC overhead was necessary for stability
|
||||
|
||||
## The Process Isolation Workaround
|
||||
|
||||
Before solid-lsp, the **only way** to use multilspy with MCP was through process isolation:
|
||||
|
||||
```python
|
||||
# Old architecture (when multilspy was still present)
|
||||
if USE_SOLID_LSP:
|
||||
mcp_factory = SerenaMCPFactorySingleProcess(context=context, project=project_file)
|
||||
else:
|
||||
# multilspy required process isolation to prevent asyncio contamination
|
||||
mcp_factory = SerenaMCPFactoryWithProcessIsolation(context=context, project=project_file)
|
||||
```
|
||||
|
||||
### Why Process Isolation Was Mandatory for Multilspy
|
||||
- **Complete Separation**: MCP server, SerenaAgent, and language servers in different processes
|
||||
- **IPC Overhead**: All communication through inter-process communication
|
||||
- **Resource Cost**: Higher memory usage and startup time
|
||||
- **Debugging Complexity**: Distributed architecture made troubleshooting harder
|
||||
|
||||
## Timeline of Changes
|
||||
|
||||
1. **Pre-solid-lsp**: Multilspy was the only option, required process isolation for MCP
|
||||
2. **Solid-lsp Introduction**: New implementation designed to eliminate asyncio issues
|
||||
3. **Current State**: Multilspy completely removed, solid-lsp is the only implementation
|
||||
4. **Process Isolation**: Now disabled by default (`USE_PROCESS_ISOLATION = False`) since it's no longer needed
|
||||
|
||||
## Key Lessons Learned
|
||||
|
||||
- **Asyncio Complexity**: Multiple event loops in the same application are extremely difficult to manage correctly
|
||||
- **MCP Integration Challenges**: MCP server's async nature requires careful consideration of async boundaries
|
||||
- **Simplicity Benefits**: Removing abstraction layers often improves stability and performance
|
||||
- **Process Isolation Trade-offs**: While effective for isolation, the performance cost is significant when not necessary
|
||||
|
||||
This migration represents a successful architectural evolution from a complex, problematic system to a simpler, more reliable one.
|
||||
@@ -0,0 +1,126 @@
|
||||
# Process Isolation: Current Status and Architecture
|
||||
|
||||
## Current Status: Disabled by Default
|
||||
|
||||
Process isolation is **disabled by default** in the current Serena implementation:
|
||||
- **Configuration**: `USE_PROCESS_ISOLATION = False` in `src/serena/constants.py:22`
|
||||
- **Reason**: No longer needed with solid-lsp architecture
|
||||
- **Default Operation**: Single process mode with MCP server, agent, and language servers in same process
|
||||
|
||||
## Why Process Isolation Is No Longer Needed
|
||||
|
||||
### Historical Context
|
||||
Process isolation was **mandatory** when using multilspy because:
|
||||
1. **Asyncio Contamination**: Multilspy leaked coroutines into MCP server's event loop
|
||||
2. **Event Loop Conflicts**: Multiple asyncio contexts caused deadlocks
|
||||
3. **Only Solution**: Complete process separation was the only way to prevent issues
|
||||
|
||||
### Current Architecture Benefits
|
||||
With solid-lsp, process isolation became **unnecessary** because:
|
||||
1. **Clean Async Boundaries**: No coroutine leakage between MCP server and language server contexts
|
||||
2. **Single Process Safety**: Solid-lsp designed to work safely within MCP server process
|
||||
3. **Performance Gains**: Direct method calls instead of expensive IPC
|
||||
|
||||
## Architecture Components (Still Present)
|
||||
|
||||
The process isolation infrastructure remains **available but unused** by default:
|
||||
|
||||
### Core Components (`src/serena/process_isolated_agent.py`)
|
||||
|
||||
#### 1. **ProcessIsolatedSerenaAgent** (lines 392-550)
|
||||
- **Purpose**: Wrapper that manages isolated agent process
|
||||
- **Communication**: Uses `multiprocessing.Pipe()` for bidirectional communication
|
||||
- **Process Management**: Creates and manages `SerenaAgentWorker` subprocess
|
||||
- **Status**: **Available but not used by default**
|
||||
|
||||
#### 2. **SerenaAgentWorker** (lines 173-389)
|
||||
- **Purpose**: Worker process hosting actual SerenaAgent
|
||||
- **Event Loop**: Polling loop checking for requests every 500ms
|
||||
- **Request Handling**: INITIALIZE, TOOL_CALL, SHUTDOWN, etc.
|
||||
- **Status**: **Available but not used by default**
|
||||
|
||||
#### 3. **ProcessIsolatedDashboard** (lines 133-170)
|
||||
- **Purpose**: Runs web dashboard in separate process
|
||||
- **Integration**: Async web server with port management
|
||||
- **Status**: **Available but not used by default**
|
||||
|
||||
### Global Synchronization (lines 25-28)
|
||||
```python
|
||||
_global_log_queue: multiprocessing.Queue = multiprocessing.Queue()
|
||||
_dashboard_ready_event = multiprocessing.Event()
|
||||
_dashboard_port_value = multiprocessing.Value("i", 0)
|
||||
global_shutdown_event = multiprocessing.Event()
|
||||
```
|
||||
**Status**: **Available but not actively used**
|
||||
|
||||
## Current MCP Factory Selection
|
||||
|
||||
In `src/serena/mcp.py:590`:
|
||||
```python
|
||||
if not USE_PROCESS_ISOLATION:
|
||||
mcp_factory = SerenaMCPFactorySingleProcess(context=context, project=project_file)
|
||||
else:
|
||||
mcp_factory = SerenaMCPFactoryWithProcessIsolation(context=context, project=project_file)
|
||||
```
|
||||
|
||||
### Default Flow
|
||||
1. **`USE_PROCESS_ISOLATION = False`** → **`SerenaMCPFactorySingleProcess`**
|
||||
2. **Single Process**: MCP server, SerenaAgent, and solid-lsp in same process
|
||||
3. **Direct Communication**: No IPC overhead, direct method calls
|
||||
|
||||
### Fallback Option
|
||||
If `USE_PROCESS_ISOLATION = True`:
|
||||
1. **`SerenaMCPFactoryWithProcessIsolation`** would be used
|
||||
2. **Separate Processes**: MCP server and SerenaAgent in different processes
|
||||
3. **IPC Communication**: Higher latency but complete isolation
|
||||
|
||||
## Benefits of Current Single Process Architecture
|
||||
|
||||
### 1. **Performance Improvements**
|
||||
- **Lower Latency**: Direct method calls vs IPC communication
|
||||
- **Reduced Memory**: No duplicate process memory footprints
|
||||
- **Faster Startup**: No process creation and initialization overhead
|
||||
|
||||
### 2. **Operational Simplicity**
|
||||
- **Unified Logging**: All components log to same destination
|
||||
- **Simpler Debugging**: Single process, unified stack traces
|
||||
- **Resource Management**: Simpler cleanup and shutdown procedures
|
||||
|
||||
### 3. **Stability Benefits**
|
||||
- **No IPC Failures**: Eliminated inter-process communication failure modes
|
||||
- **Consistent State**: No synchronization issues between processes
|
||||
- **Reliable Shutdown**: No orphaned processes or cleanup complexity
|
||||
|
||||
## When Process Isolation Might Still Be Useful
|
||||
|
||||
While not needed by default, process isolation could still be beneficial for:
|
||||
|
||||
### 1. **Fault Isolation**
|
||||
- **Agent Crashes**: Prevent agent failures from affecting MCP server
|
||||
- **Memory Protection**: Isolate memory leaks to specific processes
|
||||
- **Recovery**: Restart failed components without full system restart
|
||||
|
||||
### 2. **Resource Management**
|
||||
- **Memory Limits**: Constrain memory usage of specific components
|
||||
- **CPU Isolation**: Prevent CPU-intensive operations from blocking MCP server
|
||||
- **Security**: Additional process boundaries for security-sensitive environments
|
||||
|
||||
### 3. **Debugging and Development**
|
||||
- **Component Isolation**: Debug specific components in isolation
|
||||
- **Performance Analysis**: Measure resource usage per component
|
||||
- **Development Safety**: Prevent development errors from affecting stable components
|
||||
|
||||
## Configuration Management
|
||||
|
||||
### Enabling Process Isolation
|
||||
To re-enable process isolation:
|
||||
1. **Set**: `USE_PROCESS_ISOLATION = True` in `src/serena/constants.py`
|
||||
2. **Result**: Automatic fallback to `SerenaMCPFactoryWithProcessIsolation`
|
||||
3. **Trade-off**: Higher resource usage but complete component isolation
|
||||
|
||||
### Current Recommendation
|
||||
- **Default**: Keep `USE_PROCESS_ISOLATION = False` for optimal performance
|
||||
- **Special Cases**: Enable only when specific isolation requirements exist
|
||||
- **Testing**: Both modes should be tested to ensure compatibility
|
||||
|
||||
The current architecture successfully eliminated the need for process isolation while maintaining the capability as a fallback option for specialized use cases.
|
||||
@@ -1,16 +0,0 @@
|
||||
# Project Structure
|
||||
|
||||
The Serena codebase is organized as follows:
|
||||
|
||||
- **src/serena/**: Main package with core functionality
|
||||
- **llm/**: LLM integration
|
||||
- **util/**: Utilities
|
||||
- Key files: mcp.py, agno.py, agent.py
|
||||
- **src/multilspy/**: Language server integration
|
||||
- **language_servers/**: Implementations for different languages
|
||||
- Python (pyright/jedi)
|
||||
- Java (Eclipse JDTLS)
|
||||
- TypeScript/JavaScript
|
||||
- C#, Rust, Go, Ruby, C++
|
||||
- **test/**: Tests
|
||||
- **scripts/**: Utility scripts
|
||||
@@ -0,0 +1,173 @@
|
||||
# Serena Project Structure Overview
|
||||
|
||||
## Top-Level Organization
|
||||
|
||||
Serena is organized into several key directories, each serving distinct architectural purposes:
|
||||
|
||||
```
|
||||
serena/
|
||||
├── src/ # Main source code
|
||||
│ ├── interprompt/ # Template and prompt management system
|
||||
│ ├── serena/ # Core Serena functionality
|
||||
│ └── solidlsp/ # Language server integration layer
|
||||
├── tests/ # Test suites
|
||||
├── docs/ # Documentation
|
||||
├── prompts/ # Prompt templates and configurations
|
||||
├── contexts/ # Context definitions (desktop-app, agent, ide-assistant)
|
||||
├── modes/ # Mode definitions (planning, editing, interactive, one-shot)
|
||||
└── pyproject.toml # Project configuration and dependencies
|
||||
```
|
||||
|
||||
## Core Source Structure (`src/`)
|
||||
|
||||
### 1. **Serena Core (`src/serena/`)**
|
||||
The heart of the Serena system containing:
|
||||
|
||||
#### **Agent and Core Logic**
|
||||
- **`agent.py`**: Main `SerenaAgent` class, tool implementations, project management
|
||||
- **`mcp.py`**: Model Context Protocol server implementation and MCP factories
|
||||
- **`config.py`**: Configuration system (contexts, modes, registered configurations)
|
||||
- **`constants.py`**: System constants including `USE_PROCESS_ISOLATION = False`
|
||||
|
||||
#### **Specialized Components**
|
||||
- **`dashboard.py`**: Web dashboard for monitoring and logging (`MemoryLogHandler`, `SerenaDashboardAPI`)
|
||||
- **`symbol.py`**: Symbol management (`SymbolLocation`, `Symbol`, `SymbolManager`)
|
||||
- **`text_utils.py`**: Text processing utilities for search and file operations
|
||||
- **`gui_log_viewer.py`**: GUI log window implementation
|
||||
|
||||
#### **Integration Layers**
|
||||
- **`agno.py`**: Agno framework integration (`SerenaAgnoToolkit`, `SerenaAgnoAgentProvider`)
|
||||
- **`process_isolated_agent.py`**: Process isolation infrastructure (available but unused by default)
|
||||
- **`prompt_factory.py`**: Prompt generation and management
|
||||
|
||||
#### **Utility Modules (`src/serena/util/`)**
|
||||
- **`file_system.py`**: File scanning, gitignore parsing (`GitignoreParser`, `scan_directory`)
|
||||
- **`git.py`**: Git operations and status checking
|
||||
- **`shell.py`**: Shell command execution (`execute_shell_command`)
|
||||
- **`thread.py`**: Threading utilities with timeout support
|
||||
- **`inspection.py`**: Code inspection and language detection utilities
|
||||
|
||||
### 2. **Solid-LSP (`src/solidlsp/`)**
|
||||
Current language server integration layer:
|
||||
|
||||
#### **Core Language Server Components**
|
||||
- **`ls.py`**: Main `SolidLanguageServer` class (1,634 lines)
|
||||
- **`ls_handler.py`**: `SolidLanguageServerHandler` for protocol management (512 lines)
|
||||
- **`ls_request.py`**: Request handling (`LanguageServerRequest`)
|
||||
- **`ls_types.py`**: LSP type definitions
|
||||
- **`ls_utils.py`**: Utility classes (`TextUtils`, `PathUtils`, `FileUtils`, `SymbolUtils`)
|
||||
|
||||
#### **Protocol Handling (`src/solidlsp/lsp_protocol_handler/`)**
|
||||
- **`server.py`**: LSP protocol server implementation
|
||||
- **`lsp_requests.py`**: LSP request and notification classes
|
||||
- **`lsp_types.py`**: Complete LSP type system definitions
|
||||
- **`lsp_constants.py`**: LSP protocol constants
|
||||
|
||||
#### **Language Server Implementations (`src/solidlsp/language_servers/`)**
|
||||
Each language has its own subdirectory with specific implementations:
|
||||
- **`pyright_language_server/`**: Python support via Pyright
|
||||
- **`eclipse_jdtls/`**: Java support via Eclipse JDTLS
|
||||
- **`typescript_language_server/`**: TypeScript/JavaScript support
|
||||
- **`omnisharp/`**: C# support via OmniSharp
|
||||
- **`rust_analyzer/`**: Rust support via Rust-Analyzer
|
||||
- **`gopls/`**: Go support via Gopls
|
||||
- **`solargraph/`**: Ruby support via Solargraph
|
||||
- **`clangd_language_server/`**: C++ support via Clangd
|
||||
- **`dart_language_server/`**: Dart support
|
||||
- **`intelephense/`**: PHP support via Intelephense
|
||||
- **`kotlin_language_server/`**: Kotlin support
|
||||
|
||||
### 3. **Interprompt (`src/interprompt/`)**
|
||||
Template and prompt management system:
|
||||
|
||||
- **`multilang_prompt.py`**: Multi-language prompt templates (`MultiLangPromptTemplate`)
|
||||
- **`jinja_template.py`**: Jinja2 template integration (`JinjaTemplate`)
|
||||
- **`prompt_factory.py`**: Prompt factory base classes
|
||||
- **`util/class_decorators.py`**: Utility decorators like `@singleton`
|
||||
|
||||
## Configuration and Templates
|
||||
|
||||
### **Prompt Templates (`prompts/`)**
|
||||
Organized by context and functionality:
|
||||
- **Context-specific prompts**: Different prompt sets for different execution contexts
|
||||
- **Tool-specific prompts**: Specialized prompts for specific tools and operations
|
||||
- **Multi-language support**: Prompts available in multiple languages when applicable
|
||||
|
||||
### **Contexts (`contexts/`)**
|
||||
Execution environment definitions:
|
||||
- **`desktop-app.yml`**: Desktop application context
|
||||
- **`agent.yml`**: Agent-specific context
|
||||
- **`ide-assistant.yml`**: IDE assistant context
|
||||
- **Custom contexts**: Support for user-defined contexts
|
||||
|
||||
### **Modes (`modes/`)**
|
||||
Behavior mode definitions:
|
||||
- **`planning.yml`**: Planning mode behavior
|
||||
- **`editing.yml`**: Editing mode behavior
|
||||
- **`interactive.yml`**: Interactive mode behavior
|
||||
- **`one-shot.yml`**: One-shot execution mode
|
||||
|
||||
## Key Architectural Patterns
|
||||
|
||||
### 1. **Four-Layer Configuration Hierarchy**
|
||||
1. **Global**: `serena_config.yml`
|
||||
2. **CLI Arguments**: Runtime overrides
|
||||
3. **Project**: `.serena/project.yml`
|
||||
4. **Active Modes**: Runtime behavior modification
|
||||
|
||||
### 2. **Tool System Architecture**
|
||||
- **Base Classes**: `ToolInterface`, `Tool` in `agent.py`
|
||||
- **Marker Interfaces**: `ToolMarkerCanEdit`, `ToolMarkerDoesNotRequireActiveProject`
|
||||
- **Tool Registry**: `ToolRegistry` with automatic tool discovery via `_iter_tool_classes()`
|
||||
- **Tool Categories**: Semantic tools, file operations, project management, meta-tools
|
||||
|
||||
### 3. **Integration Patterns**
|
||||
- **MCP Server**: Primary integration via `SerenaMCPFactory` classes
|
||||
- **Agno Agent**: Model-agnostic integration via `SerenaAgnoToolkit`
|
||||
- **Process Isolation**: Optional isolation via `ProcessIsolatedSerenaAgent`
|
||||
|
||||
### 4. **Memory Management**
|
||||
- **Project Memories**: `.serena/memories/` directory for project-specific information
|
||||
- **Memory Tools**: `WriteMemoryTool`, `ReadMemoryTool`, `ListMemoriesTool`
|
||||
- **Memory Managers**: `MemoriesManager`, `MemoriesManagerMDFilesInProject`
|
||||
|
||||
## Dependencies and Build System
|
||||
|
||||
### **Development Tools**
|
||||
- **`uv`**: Dependency management and virtual environment
|
||||
- **`poe`**: Task orchestration (defined in `pyproject.toml`)
|
||||
- **Essential Commands**: `uv run poe lint`, `uv run poe format`, `uv run poe type-check`, `uv run poe test`
|
||||
|
||||
### **Key Dependencies**
|
||||
- **Language Server Protocol**: LSP implementations for each supported language
|
||||
- **MCP (Model Context Protocol)**: For integration with Claude Desktop and other MCP clients
|
||||
- **Agno Framework**: For model-agnostic agent implementations
|
||||
- **AsyncIO**: For concurrent operations (carefully managed to avoid contamination)
|
||||
|
||||
## Project State Management
|
||||
|
||||
### **Project Configuration**
|
||||
- **Global Config**: `serena_config.yml` in user home directory
|
||||
- **Project Config**: `.serena/project.yml` in project root
|
||||
- **Auto-generation**: Automatic creation of default configurations when missing
|
||||
|
||||
### **Language Detection**
|
||||
- **Automatic**: Based on file composition analysis
|
||||
- **Manual Override**: Via project configuration
|
||||
- **Multi-language**: Support for projects with multiple languages
|
||||
|
||||
## Runtime Architecture
|
||||
|
||||
### **Default Operation Mode**
|
||||
- **Single Process**: MCP server, agent, and language servers in same process
|
||||
- **Direct Communication**: No IPC overhead
|
||||
- **Solid-LSP**: Only language server implementation
|
||||
- **Process Isolation**: Disabled by default (`USE_PROCESS_ISOLATION = False`)
|
||||
|
||||
### **Semantic Tool Integration**
|
||||
- **Symbol-based Operations**: `find_symbol`, `replace_symbol_body`, `insert_after_symbol`
|
||||
- **Regex-based Operations**: `replace_regex` for fine-grained edits
|
||||
- **File Operations**: `read_file`, `create_text_file`, `list_dir`
|
||||
- **Project Operations**: `search_for_pattern`, `get_symbols_overview`
|
||||
|
||||
This structure reflects Serena's evolution from a complex, multi-process system to a streamlined, single-process architecture that maintains all semantic capabilities while improving performance and reliability.
|
||||
@@ -1,161 +0,0 @@
|
||||
# Solid-LSP: Deadlock-Free Language Server Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
Solid-LSP (`src/solidlsp/`) is a **simplified, deadlock-resistant** reimplementation of the language server communication layer, designed to replace the problematic `multilspy` architecture that suffered from asyncio contamination issues when used with MCP servers.
|
||||
|
||||
## The Problem Solid-LSP Solves
|
||||
|
||||
### Root Cause: Asyncio Context Contamination
|
||||
The original `multilspy` implementation created **asyncio deadlocks** in MCP environments due to:
|
||||
|
||||
1. **Dual Event Loop Interference**: MCP server runs its own asyncio event loop, while SerenaAgent creates separate asyncio loops for language servers and dashboard
|
||||
2. **Coroutine Leakage**: When MCP server calls SerenaAgent tools, coroutines created by `multilspy.LanguageServer.request_full_symbol_tree()` were leaked into the MCP server's asyncio context but never awaited
|
||||
3. **Resource Exhaustion**: Accumulated unawaited coroutines in the MCP server's event loop eventually caused blocking and apparent "hangs"
|
||||
4. **MCP-Specific Issue**: Only occurred when using MCP clients, not in direct script execution
|
||||
|
||||
### Evidence from Logs
|
||||
```
|
||||
INFO mcp.server.lowlevel.server:_handle_message:524 - Warning: RuntimeWarning: coroutine 'LanguageServer.request_full_symbol_tree' was never awaited
|
||||
```
|
||||
|
||||
## Solid-LSP Architecture
|
||||
|
||||
### Core Design Principles
|
||||
|
||||
1. **Simplified Protocol Handling**: Eliminates the complex `lsp_protocol_handler` layer from `multilspy`
|
||||
2. **Direct Async Management**: More explicit control over async operations without complex threading abstractions
|
||||
3. **Single Process Operation**: Designed to work directly in the MCP server process without requiring process isolation
|
||||
4. **Clean Asyncio Boundaries**: Prevents coroutine leakage between async contexts
|
||||
|
||||
### Key Architectural Differences from multilspy
|
||||
|
||||
#### 1. Simplified Handler Architecture
|
||||
- **multilspy**: Complex `LanguageServerHandler` with extensive async task management and event loops
|
||||
- **solid-lsp**: Streamlined `SolidLanguageServerHandler` with direct process management
|
||||
|
||||
#### 2. Protocol Simplification
|
||||
- **No separate protocol handler package**: Direct LSP communication without abstraction layers
|
||||
- **Reduced complexity**: Fewer components and dependencies
|
||||
- **Same language server support**: Identical `language_servers/` implementations for all supported languages
|
||||
|
||||
#### 3. Enhanced Process Control
|
||||
**solid-lsp** provides additional methods for finer process control:
|
||||
- `_start_server_process()`: Direct process control
|
||||
- `_start_server()`: Server initialization
|
||||
- `_server_context` attribute: Enhanced server state management
|
||||
|
||||
## Process Isolation: No Longer Required
|
||||
|
||||
### Key Benefit: Single Process Operation
|
||||
|
||||
With solid-lsp, **process isolation is no longer needed** and is **disabled by default**:
|
||||
|
||||
```python
|
||||
# src/serena/mcp.py:581-585
|
||||
if USE_SOLID_LSP:
|
||||
mcp_factory = SerenaMCPFactorySingleProcess(context=context, project=project_file)
|
||||
else:
|
||||
# using multilspy requires process isolation to prevent asyncio contamination
|
||||
mcp_factory = SerenaMCPFactoryWithProcessIsolation(context=context, project=project_file)
|
||||
```
|
||||
|
||||
### Benefits of Single Process Operation
|
||||
1. **Lower Resource Usage**: No separate processes for SerenaAgent and language servers
|
||||
2. **Reduced Latency**: Direct method calls instead of inter-process communication
|
||||
3. **Simplified Architecture**: No IPC overhead or process management complexity
|
||||
4. **Better Performance**: Eliminated serialization/deserialization overhead
|
||||
5. **Easier Debugging**: All components in same process, simplified stack traces
|
||||
|
||||
### Why multilspy Required Process Isolation
|
||||
- **Asyncio Contamination**: multilspy's async patterns leaked coroutines into MCP server context
|
||||
- **Mandatory Isolation**: Process boundaries were the only way to prevent deadlocks
|
||||
- **Performance Penalty**: IPC overhead was necessary for stability
|
||||
|
||||
### How solid-lsp Enables Single Process
|
||||
- **Clean Async Patterns**: No coroutine leakage between MCP server and language server contexts
|
||||
- **Direct Integration**: Can safely run in same process as MCP server
|
||||
- **Stable Operation**: No more timeout issues or hanging operations
|
||||
|
||||
## Supported Language Servers
|
||||
|
||||
Solid-LSP maintains compatibility with all language servers from `multilspy`:
|
||||
|
||||
### Primary Languages
|
||||
- **Python**: Pyright Language Server (`src/solidlsp/language_servers/pyright_language_server/`)
|
||||
- **Java**: Eclipse JDTLS (`src/solidlsp/language_servers/eclipse_jdtls/`)
|
||||
- **TypeScript/JavaScript**: TypeScript Language Server (`src/solidlsp/language_servers/typescript_language_server/`)
|
||||
|
||||
### Additional Languages
|
||||
- **C#**: OmniSharp (`src/solidlsp/language_servers/omnisharp/`)
|
||||
- **Rust**: Rust-Analyzer (`src/solidlsp/language_servers/rust_analyzer/`)
|
||||
- **Go**: Gopls (`src/solidlsp/language_servers/gopls/`)
|
||||
- **Ruby**: Solargraph (`src/solidlsp/language_servers/solargraph/`)
|
||||
- **C++**: Clangd (`src/solidlsp/language_servers/clangd_language_server/`)
|
||||
- **Dart**: Dart Language Server (`src/solidlsp/language_servers/dart_language_server/`)
|
||||
- **PHP**: Intelephense (`src/solidlsp/language_servers/intelephense/`)
|
||||
- **Kotlin**: Kotlin Language Server (`src/solidlsp/language_servers/kotlin_language_server/`)
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Core Classes
|
||||
|
||||
#### `SolidLanguageServer` (`src/solidlsp/ls.py`)
|
||||
- **1,634 lines**: Main language server interface
|
||||
- **Enhanced methods**: Same semantic capabilities as `multilspy.LanguageServer` but with improved async handling
|
||||
- **Lifecycle control**: `start()`, `stop()`, `is_running()`, `language_server()` property (similar to multilspy's `SyncLanguageServer`)
|
||||
|
||||
#### `SolidLanguageServerHandler` (`src/solidlsp/ls_handler.py`)
|
||||
- **512 lines**: Simplified LSP protocol handler
|
||||
- **Direct process management**: More straightforward than `multilspy`'s complex async task orchestration
|
||||
- **Improved cleanup**: Better resource management and process termination
|
||||
|
||||
#### `SolidLspRequest` (`src/solidlsp/lsp_request.py`)
|
||||
- **Simplified request handling**: Direct LSP request/response without complex protocol abstractions
|
||||
|
||||
### Configuration Integration
|
||||
|
||||
Solid-LSP is controlled by the `USE_SOLID_LSP` constant in `src/serena/constants.py`:
|
||||
|
||||
```python
|
||||
USE_SOLID_LSP = True # Default: enabled
|
||||
```
|
||||
|
||||
This allows:
|
||||
- **Default Operation**: solid-lsp enabled by default for stability
|
||||
- **Fallback Capability**: Can switch back to multilspy + process isolation if needed
|
||||
- **A/B Testing**: Easy comparison between implementations
|
||||
|
||||
## Deadlock Prevention
|
||||
|
||||
### How Solid-LSP Prevents Deadlocks
|
||||
|
||||
1. **Simplified Async Patterns**: Reduces complexity that led to coroutine leakage
|
||||
2. **Clean Boundaries**: Proper async context management prevents MCP server contamination
|
||||
3. **Direct Operations**: Eliminates abstraction layers where async context could be corrupted
|
||||
4. **Single Process Safety**: Designed to work safely within MCP server process
|
||||
|
||||
### Verification
|
||||
|
||||
The solid-lsp approach provides:
|
||||
- **No more timeout issues**: Clean async boundaries prevent MCP server contamination
|
||||
- **Stable MCP operation**: Language server calls no longer hang after first timeout
|
||||
- **Resource efficiency**: Lower memory usage without separate processes
|
||||
- **Reliable semantic tools**: `find_symbol` and other tools work consistently
|
||||
- **Better Performance**: Direct method calls instead of IPC
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Current Status
|
||||
- **Default**: Solid-LSP is the default implementation (`USE_SOLID_LSP = True`)
|
||||
- **Stable**: Resolves known deadlock issues in MCP environments
|
||||
- **Compatible**: Supports all existing language servers and semantic operations
|
||||
- **Performant**: Single process operation with lower overhead
|
||||
|
||||
### Rollback Path
|
||||
If issues arise, can fallback to multilspy with process isolation:
|
||||
1. **Disable solid-lsp**: Set `USE_SOLID_LSP = False`
|
||||
2. **Automatic fallback**: System automatically uses multilspy + process isolation
|
||||
3. **Performance trade-off**: Higher resource usage but guaranteed isolation
|
||||
|
||||
Solid-LSP represents an **architectural evolution** that eliminates the need for process isolation while maintaining all semantic capabilities and providing **better performance and stability** for MCP server deployments.
|
||||
@@ -0,0 +1,97 @@
|
||||
# Supported Language Servers in Solid-LSP
|
||||
|
||||
Serena currently supports multiple programming languages through language servers via the solid-lsp architecture located in `src/solidlsp/language_servers/`.
|
||||
|
||||
## Primary Languages (Directly Supported)
|
||||
|
||||
### Python
|
||||
- **Language Server**: Pyright Language Server
|
||||
- **Location**: `src/solidlsp/language_servers/pyright_language_server/`
|
||||
- **Features**: Full semantic analysis, type checking, symbol navigation
|
||||
- **Previously**: Also supported Jedi (now deprecated in favor of Pyright)
|
||||
|
||||
### Java
|
||||
- **Language Server**: Eclipse JDTLS (Java Development Tools Language Server)
|
||||
- **Location**: `src/solidlsp/language_servers/eclipse_jdtls/`
|
||||
- **Features**: Full Java language support, Maven/Gradle integration
|
||||
- **Note**: Startup can be slow, especially on initial launch
|
||||
|
||||
### TypeScript/JavaScript
|
||||
- **Language Server**: TypeScript Language Server
|
||||
- **Location**: `src/solidlsp/language_servers/typescript_language_server/`
|
||||
- **Features**: TypeScript and JavaScript support, type checking, IntelliSense
|
||||
|
||||
## Additional Supported Languages
|
||||
|
||||
### C#
|
||||
- **Language Server**: OmniSharp
|
||||
- **Location**: `src/solidlsp/language_servers/omnisharp/`
|
||||
- **Features**: Full C# language support, .NET integration
|
||||
|
||||
### Rust
|
||||
- **Language Server**: Rust-Analyzer
|
||||
- **Location**: `src/solidlsp/language_servers/rust_analyzer/`
|
||||
- **Features**: Rust language support, cargo integration
|
||||
|
||||
### Go
|
||||
- **Language Server**: Gopls
|
||||
- **Location**: `src/solidlsp/language_servers/gopls/`
|
||||
- **Features**: Go language support, module system integration
|
||||
|
||||
### Ruby
|
||||
- **Language Server**: Solargraph
|
||||
- **Location**: `src/solidlsp/language_servers/solargraph/`
|
||||
- **Features**: Ruby language support, gem integration
|
||||
|
||||
### C++
|
||||
- **Language Server**: Clangd
|
||||
- **Location**: `src/solidlsp/language_servers/clangd_language_server/`
|
||||
- **Features**: C++ language support, CMake integration
|
||||
|
||||
### Dart
|
||||
- **Language Server**: Dart Language Server
|
||||
- **Location**: `src/solidlsp/language_servers/dart_language_server/`
|
||||
- **Features**: Dart and Flutter support
|
||||
|
||||
### PHP
|
||||
- **Language Server**: Intelephense
|
||||
- **Location**: `src/solidlsp/language_servers/intelephense/`
|
||||
- **Features**: PHP language support, Composer integration
|
||||
|
||||
### Kotlin
|
||||
- **Language Server**: Kotlin Language Server
|
||||
- **Location**: `src/solidlsp/language_servers/kotlin_language_server/`
|
||||
- **Features**: Kotlin language support, Gradle integration
|
||||
|
||||
## Language Server Protocol (LSP) Integration
|
||||
|
||||
All language servers utilize the Language Server Protocol (LSP) to provide:
|
||||
- **Semantic Analysis**: Symbol definitions, references, and relationships
|
||||
- **Code Navigation**: Go-to-definition, find-references, symbol search
|
||||
- **Error Detection**: Syntax and semantic error reporting
|
||||
- **Code Completion**: IntelliSense-style code completion
|
||||
- **Documentation**: Hover information and signature help
|
||||
|
||||
## Architecture Integration
|
||||
|
||||
### Solid-LSP Framework
|
||||
- **Unified Interface**: All language servers use the same `SolidLanguageServer` interface
|
||||
- **Configuration**: `LanguageServerConfig` handles language-specific settings
|
||||
- **Lifecycle Management**: Start, stop, and restart capabilities for all servers
|
||||
- **Process Management**: Direct process control for each language server
|
||||
|
||||
### Project Integration
|
||||
- **Automatic Detection**: Language selection based on project file composition
|
||||
- **Configuration**: Project-specific language server settings via `.serena/project.yml`
|
||||
- **Ignored Paths**: Respects gitignore and custom ignore patterns
|
||||
- **Timeout Settings**: Configurable per-language server timeouts
|
||||
|
||||
## Adding New Language Servers
|
||||
|
||||
The architecture supports adding new language servers by:
|
||||
1. **Creating Language Server Class**: Implement language-specific server in `src/solidlsp/language_servers/`
|
||||
2. **Configuration**: Add language detection and configuration rules
|
||||
3. **Integration**: Register with the solid-lsp framework
|
||||
4. **Testing**: Ensure semantic tools work correctly with the new language server
|
||||
|
||||
The system is designed to be extensible while maintaining consistent behavior across all supported languages.
|
||||
Reference in New Issue
Block a user