mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-04 12:13:05 +00:00
Alex
94bc4d1eb6
* fix: route remote-device tool through Redis so scheduled runs reach the device
The remote-device tool worked interactively but timed out on every scheduled
run. DeviceBroker was an in-process, in-memory singleton, but scheduled runs
execute in the Celery worker — a different process from the gunicorn web tier
that holds the device's SSE session — so a worker-side dispatch never reached
the device and the tool always hit its deadline.
Make the broker Redis-backed so every hop crosses the process boundary:
- queued commands -> Redis list dev:cmd:{device_id}
- output chunks -> Redis stream dev:out:{invocation_id}
- invocation metadata -> Redis hash dev:inv:{invocation_id}
- SSE upgrade tickets -> Redis key dev🎫{device_id}
Per-connection SSE session state stays in the web process. Reuses the existing
get_redis_instance()/CACHE_REDIS_URL; no new infrastructure. Also makes the web
tier safe to scale past one worker.
Concurrency hardening (from adversarial review + real-Redis e2e):
- XADD the output/control chunk before flipping completed=1, and have
drain_output do a final non-blocking flush after observing completion, so a
reader can't see completion and stop before the control chunk lands (this had
reintroduced the false "device did not respond (timed out)" under a race).
- _collect_result builds the result from drained chunks, checks the deadline
only after capturing a chunk, and falls back to the authoritative snapshot
(before cleanup) when no control chunk was observed.
- Audit outcome is written from locally-known fields so it survives the worker
racing to delete the invocation; a denied command now records a terminal
"denied" outcome instead of staying "dispatched".
- cmd-queue TTL raised to 900s (>= max drain deadline); dispatch-failure and
reaped-invocation cleanup; UTF-8 byte counts.
Tests: new tests/devices/{conftest (FakeRedis double), test_broker_cross_process,
test_broker_race, test_submit_output_audit}; drain/cleanup/ticket tests rewritten
for the Redis contract. The race tests fail against the pre-fix code. ruff clean;
device + tool-executor suites green.
* fix: log instead of silently passing on failed-dispatch cleanup
Addresses the code-quality lint on the best-effort hash delete in
dispatch_invocation's failure path: replace the bare `except: pass` with a
logger.debug carrying the invocation_id. No behavior change — cleanup stays
best-effort and still returns a failed Invocation.
67 lines
2.4 KiB
Python
67 lines
2.4 KiB
Python
"""Tests for ``DeviceBroker.drain_output`` deadline + control handling."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import time
|
|
|
|
|
|
def _dispatch(broker):
|
|
envelope = {"invocation_id": "inv_test", "action": "run_command"}
|
|
return broker.dispatch_invocation("dev_x", "user_x", envelope)
|
|
|
|
|
|
def test_drain_output_returns_after_deadline_with_no_output(broker_env):
|
|
# Device never connects / posts output: the generator must stop once the
|
|
# deadline passes instead of looping forever.
|
|
broker, _fake = broker_env
|
|
inv = _dispatch(broker)
|
|
deadline = time.time() + 0.2
|
|
start = time.time()
|
|
chunks = list(
|
|
broker.drain_output(inv.invocation_id, timeout=0.05, deadline=deadline)
|
|
)
|
|
elapsed = time.time() - start
|
|
assert chunks == []
|
|
assert elapsed < 2.0
|
|
|
|
|
|
def test_drain_output_yields_then_stops_on_control(broker_env):
|
|
# A normal stream drains stdout then stops on the closing control chunk.
|
|
broker, _fake = broker_env
|
|
inv = _dispatch(broker)
|
|
broker.submit_output_chunk(inv.invocation_id, {"stream": "stdout", "chunk": "hello"})
|
|
broker.submit_output_chunk(
|
|
inv.invocation_id, {"stream": "control", "exit_code": 0, "duration_ms": 1}
|
|
)
|
|
deadline = time.time() + 5.0
|
|
chunks = list(
|
|
broker.drain_output(inv.invocation_id, timeout=0.05, deadline=deadline)
|
|
)
|
|
assert chunks[0] == {"stream": "stdout", "chunk": "hello"}
|
|
assert chunks[-1]["stream"] == "control"
|
|
assert chunks[-1]["exit_code"] == 0
|
|
|
|
|
|
def test_drain_output_stops_when_completed_without_control(broker_env):
|
|
# If the invocation is marked completed but no control chunk is on the
|
|
# stream, the completed flag still terminates the drain loop.
|
|
broker, fake = broker_env
|
|
inv = _dispatch(broker)
|
|
fake.hset(f"dev:inv:{inv.invocation_id}", mapping={"completed": "1"})
|
|
chunks = list(broker.drain_output(inv.invocation_id, timeout=0.05))
|
|
assert chunks == []
|
|
|
|
|
|
def test_drain_output_reports_error_when_redis_unavailable(monkeypatch):
|
|
# No Redis: the tool must get a clear control/error chunk, not hang.
|
|
from application.devices.broker import DeviceBroker
|
|
|
|
monkeypatch.setattr(
|
|
"application.devices.broker.get_redis_instance", lambda: None
|
|
)
|
|
broker = DeviceBroker()
|
|
chunks = list(broker.drain_output("inv_missing", timeout=0.05))
|
|
assert len(chunks) == 1
|
|
assert chunks[0]["stream"] == "control"
|
|
assert "unavailable" in chunks[0]["error"]
|