mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-05 08:14:32 +00:00
Source access control --------------------- `active_docs` is client-supplied and reached the retriever unchecked, and the retriever queries `WHERE source_id = <id>` with no owner predicate — so any caller could pass any source id to /stream or /api/answer and have another tenant's documents quoted back, while /api/sources/<id>/search correctly refused the same id. Gate it through `can_access`, the helper the guarded endpoints already use, and filter `self.source` down to the authorized set. Fails closed: no principal, or a check that errors, drops the source. Three sibling paths had the same gap: - workflow agent nodes: `AgentNodeConfig.sources` is written verbatim from client JSON at save time and nothing validated it, so a node could name any tenant's source. Gate against the workflow owner, so shared workflows keep reading their owner's sources like shared agents do. - /api/share: `_resolve_source_pg_id` resolved any id with no ownership predicate and baked it into the agent the share creates; /api/search then searched it. Authorize before attaching. - search_service: re-resolve the ids stored on an agent row instead of trusting them, so a row written by any future path with the same gap cannot be read back. Team grantees previously lost their source's retrieval config: the post-check read was still owner-scoped, so it missed and fell back to defaults (an `agentic_tool` source was bulk-prefetched for every grantee). Read unscoped after `can_access` passes. Retrieval --------- `PGVectorStore._ensure_table_exists` created an IVFFlat index on the empty table it had just created. IVFFlat computes centroids at build time, so those centroids were random, and combined with the `source_id` post-filter a source with hundreds of embedded chunks returned zero rows — retrieval reported no documents, the model answered from memory, and nothing was logged. Stop creating the index (exact search is correct and fast well past the sizes most deployments reach); raise `ivfflat.probes` to sqrt(lists) where an index still exists; and re-run a short indexed search exactly, since post-filtering means no index setting can guarantee a full result. `graphrag` had the same empty-table index with no fallback at all. Also: bound `chunks` to 0-500 on both the request and agent paths (0 still means "skip retrieval"), let a source's configured `retrieval.chunks` outrank the request body, and cap ClassicRAG's per-source floor at max(top_k, n_sources) so attaching sources cannot inflate the result set. Silent failures --------------- An empty retrieval was invisible to both the model and the client: the `source` event was suppressed when the list was empty, so "searched and found nothing" looked identical to "no source attached", and the prompt said nothing at all. Emit the event always, and tell the model when a search ran and returned nothing. A file that parses to nothing now fails ingest with a message naming the cause instead of storing an embedding of the empty string. `score_threshold` returns warnings when the active store or retriever cannot honour it. Prompt structure ---------------- Retrieved documents move from the system prompt into the user turn, with the injection guard restated next to them: they change every turn (defeating prefix caching), they are third-party text that should not carry system authority, and routing them through the query budget makes them truncatable rather than silently crowding it out. Documents are shed lowest-ranked-first before the question is touched. The six chat presets (3 tones x 2 retrieval modes) differed only in their Answering section; they are now composed from single-source fragments at load time, not through Jinja inheritance, which would have opened a file-read surface in the template sandbox and broken the tool-prefetch parser. Per-tool guidance moves out of the prompt into tool schemas, so it travels with the tool and cannot render when the tool is absent. A plain-text custom prompt is staged as a persona value inside the skeleton instead of replacing it wholesale — it used to silently lose the injection guard, platform block, memory and attachments, and its braces are now inert. Other fixes ----------- - agents/base: an oversized system prompt drove the query budget negative and dispatched a full-price request with an empty question; raise instead. - llm/anthropic: migrate off the retired Text Completions API. It flattened history to first+last message and ignored tools entirely. Adds the missing Anthropic handler, without which every tool call was silently dropped. - sources/upload: `sitemap` had no branch, so every sitemap ingest died on a TypeError; `validate_url` now rejects a falsy URL cleanly. - workflow nodes: retrieved documents never reached the node agent, so a classic node with a source and an ordinary prompt answered "I have no documents" while the run reported completed. - parser/bulk: copy the metadata dict, or every chunk reports the last chunk's token_count. - crawler_loader: carry the page title, or citations render the whole chunk body as the label.
394 lines
16 KiB
Python
394 lines
16 KiB
Python
"""Conversation sharing routes."""
|
|
|
|
import logging
|
|
import uuid
|
|
|
|
from flask import current_app, jsonify, make_response, request
|
|
from flask_restx import fields, inputs, Namespace, Resource
|
|
from sqlalchemy import text as _sql_text
|
|
|
|
from application.api import api
|
|
from application.storage.db.base_repository import looks_like_uuid
|
|
from application.storage.db.repositories.agents import AgentsRepository
|
|
from application.storage.db.repositories.attachments import AttachmentsRepository
|
|
from application.storage.db.repositories.conversations import ConversationsRepository
|
|
from application.storage.db.repositories.shared_conversations import (
|
|
SharedConversationsRepository,
|
|
)
|
|
from application.storage.db.session import db_readonly, db_session
|
|
from application.utils import check_required_fields
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
sharing_ns = Namespace(
|
|
"sharing", description="Conversation sharing operations", path="/api"
|
|
)
|
|
|
|
|
|
def _resolve_prompt_pg_id(conn, prompt_id_raw, user_id):
|
|
"""Translate an incoming prompt id (UUID or legacy Mongo ObjectId) to a PG UUID.
|
|
|
|
Scoped by ``user_id`` so a caller can't link another user's prompt
|
|
into their share record. Returns ``None`` for sentinel values
|
|
(``"default"``) or unresolved ids.
|
|
"""
|
|
if not prompt_id_raw or prompt_id_raw == "default":
|
|
return None
|
|
value = str(prompt_id_raw)
|
|
# Already UUID — trust it but still require ownership. A shape-gate
|
|
# (rather than a loose ``len == 36 and '-' in value`` check) keeps
|
|
# non-UUID input out of ``CAST(:pid AS uuid)``; the cast would raise
|
|
# and poison the readonly transaction otherwise.
|
|
if looks_like_uuid(value):
|
|
row = conn.execute(
|
|
_sql_text(
|
|
"SELECT id FROM prompts WHERE id = CAST(:pid AS uuid) "
|
|
"AND user_id = :uid"
|
|
),
|
|
{"pid": value, "uid": user_id},
|
|
).fetchone()
|
|
return str(row[0]) if row else None
|
|
# Legacy Mongo ObjectId fallback.
|
|
row = conn.execute(
|
|
_sql_text(
|
|
"SELECT id FROM prompts WHERE legacy_mongo_id = :pid "
|
|
"AND user_id = :uid"
|
|
),
|
|
{"pid": value, "uid": user_id},
|
|
).fetchone()
|
|
return str(row[0]) if row else None
|
|
|
|
|
|
def _resolve_source_pg_id(conn, source_raw, user=None):
|
|
"""Translate a source id (UUID or legacy Mongo ObjectId) to a PG UUID.
|
|
|
|
``source_raw`` is client-supplied and was resolved with no ownership
|
|
predicate, so a caller could bake any tenant's source id into the agent
|
|
this share creates — and ``/api/search`` then searched it. ``user`` is
|
|
required to authorize; without it nothing resolves.
|
|
"""
|
|
if not source_raw:
|
|
return None
|
|
if not user:
|
|
logger.warning("Refusing to resolve a share source with no principal.")
|
|
return None
|
|
value = str(source_raw)
|
|
# See ``_resolve_prompt_pg_id`` for the shape-gate rationale.
|
|
if looks_like_uuid(value):
|
|
row = conn.execute(
|
|
_sql_text(
|
|
"SELECT id FROM sources WHERE id = CAST(:sid AS uuid)"
|
|
),
|
|
{"sid": value},
|
|
).fetchone()
|
|
return _authorized_source(conn, row, user)
|
|
row = conn.execute(
|
|
_sql_text("SELECT id FROM sources WHERE legacy_mongo_id = :sid"),
|
|
{"sid": value},
|
|
).fetchone()
|
|
return _authorized_source(conn, row, user)
|
|
|
|
|
|
def _authorized_source(conn, row, user):
|
|
"""Return the resolved source id only if ``user`` may reference it."""
|
|
if not row:
|
|
return None
|
|
resolved = str(row[0])
|
|
from application.api.user.team_sharing import can_access
|
|
|
|
if not can_access(conn, "source", resolved, user):
|
|
logger.warning(
|
|
"Refusing to attach source %s to a share: %s has no access.",
|
|
resolved, user,
|
|
)
|
|
return None
|
|
return resolved
|
|
|
|
|
|
def _find_reusable_share_agent(
|
|
conn, user_id, *, prompt_pg_id, chunks, source_pg_id, retriever,
|
|
):
|
|
"""Find an existing share-as-agent key row matching these parameters.
|
|
|
|
Mirrors the legacy Mongo ``agents_collection.find_one`` pre-existence
|
|
check. Used to reuse an api key across repeated shares of the same
|
|
conversation with the same prompt/chunks/source/retriever.
|
|
"""
|
|
clauses = ["user_id = :uid", "key IS NOT NULL"]
|
|
params: dict = {"uid": user_id}
|
|
if prompt_pg_id is None:
|
|
clauses.append("prompt_id IS NULL")
|
|
else:
|
|
clauses.append("prompt_id = CAST(:pid AS uuid)")
|
|
params["pid"] = prompt_pg_id
|
|
if chunks is None:
|
|
clauses.append("chunks IS NULL")
|
|
else:
|
|
clauses.append("chunks = :chunks")
|
|
params["chunks"] = int(chunks)
|
|
if source_pg_id is None:
|
|
clauses.append("source_id IS NULL")
|
|
else:
|
|
clauses.append("source_id = CAST(:sid AS uuid)")
|
|
params["sid"] = source_pg_id
|
|
if retriever is None:
|
|
clauses.append("retriever IS NULL")
|
|
else:
|
|
clauses.append("retriever = :retr")
|
|
params["retr"] = retriever
|
|
sql = (
|
|
"SELECT * FROM agents WHERE "
|
|
+ " AND ".join(clauses)
|
|
+ " LIMIT 1"
|
|
)
|
|
row = conn.execute(_sql_text(sql), params).fetchone()
|
|
if row is None:
|
|
return None
|
|
mapping = dict(row._mapping)
|
|
mapping["id"] = str(mapping["id"]) if mapping.get("id") else None
|
|
return mapping
|
|
|
|
|
|
@sharing_ns.route("/share")
|
|
class ShareConversation(Resource):
|
|
share_conversation_model = api.model(
|
|
"ShareConversationModel",
|
|
{
|
|
"conversation_id": fields.String(
|
|
required=True, description="Conversation ID"
|
|
),
|
|
"user": fields.String(description="User ID (optional)"),
|
|
"prompt_id": fields.String(description="Prompt ID (optional)"),
|
|
"chunks": fields.Integer(description="Chunks count (optional)"),
|
|
},
|
|
)
|
|
|
|
@api.expect(share_conversation_model)
|
|
@api.doc(description="Share a conversation")
|
|
def post(self):
|
|
decoded_token = request.decoded_token
|
|
if not decoded_token:
|
|
return make_response(jsonify({"success": False}), 401)
|
|
user = decoded_token.get("sub")
|
|
data = request.get_json()
|
|
required_fields = ["conversation_id"]
|
|
missing_fields = check_required_fields(data, required_fields)
|
|
if missing_fields:
|
|
return missing_fields
|
|
is_promptable = request.args.get("isPromptable", type=inputs.boolean)
|
|
if is_promptable is None:
|
|
return make_response(
|
|
jsonify({"success": False, "message": "isPromptable is required"}), 400
|
|
)
|
|
conversation_id = data["conversation_id"]
|
|
|
|
try:
|
|
with db_session() as conn:
|
|
conv_repo = ConversationsRepository(conn)
|
|
shared_repo = SharedConversationsRepository(conn)
|
|
agents_repo = AgentsRepository(conn)
|
|
|
|
conversation = conv_repo.get_any(conversation_id, user)
|
|
if conversation is None:
|
|
return make_response(
|
|
jsonify(
|
|
{
|
|
"status": "error",
|
|
"message": "Conversation does not exist",
|
|
}
|
|
),
|
|
404,
|
|
)
|
|
conv_pg_id = str(conversation["id"])
|
|
current_n_queries = conv_repo.message_count(conv_pg_id)
|
|
|
|
if is_promptable:
|
|
prompt_id_raw = data.get("prompt_id", "default")
|
|
chunks_raw = data.get("chunks", "2")
|
|
try:
|
|
chunks_int = int(chunks_raw) if chunks_raw not in (None, "") else None
|
|
except (TypeError, ValueError):
|
|
chunks_int = None
|
|
|
|
prompt_pg_id = _resolve_prompt_pg_id(conn, prompt_id_raw, user)
|
|
source_pg_id = _resolve_source_pg_id(
|
|
conn, data.get("source"), user
|
|
)
|
|
retriever = data.get("retriever")
|
|
|
|
reusable = _find_reusable_share_agent(
|
|
conn, user,
|
|
prompt_pg_id=prompt_pg_id,
|
|
chunks=chunks_int,
|
|
source_pg_id=source_pg_id,
|
|
retriever=retriever,
|
|
)
|
|
if reusable:
|
|
api_uuid = reusable.get("key")
|
|
else:
|
|
api_uuid = str(uuid.uuid4())
|
|
name = (conversation.get("name") or "") + "(shared)"
|
|
agents_repo.create(
|
|
user,
|
|
name,
|
|
"published",
|
|
key=api_uuid,
|
|
retriever=retriever,
|
|
chunks=chunks_int,
|
|
prompt_id=prompt_pg_id,
|
|
source_id=source_pg_id,
|
|
)
|
|
|
|
share = shared_repo.get_or_create(
|
|
conv_pg_id,
|
|
user,
|
|
is_promptable=True,
|
|
first_n_queries=current_n_queries,
|
|
api_key=api_uuid,
|
|
prompt_id=prompt_pg_id,
|
|
chunks=chunks_int,
|
|
)
|
|
return make_response(
|
|
jsonify(
|
|
{
|
|
"success": True,
|
|
"identifier": str(share["uuid"]),
|
|
}
|
|
),
|
|
201 if reusable is None else 200,
|
|
)
|
|
|
|
# Non-promptable share path.
|
|
share = shared_repo.get_or_create(
|
|
conv_pg_id,
|
|
user,
|
|
is_promptable=False,
|
|
first_n_queries=current_n_queries,
|
|
api_key=None,
|
|
)
|
|
return make_response(
|
|
jsonify(
|
|
{
|
|
"success": True,
|
|
"identifier": str(share["uuid"]),
|
|
}
|
|
),
|
|
201,
|
|
)
|
|
except Exception as err:
|
|
current_app.logger.error(
|
|
f"Error sharing conversation: {err}", exc_info=True
|
|
)
|
|
return make_response(jsonify({"success": False}), 400)
|
|
|
|
|
|
@sharing_ns.route("/shared_conversation/<string:identifier>")
|
|
class GetPubliclySharedConversations(Resource):
|
|
@api.doc(description="Get publicly shared conversations by identifier")
|
|
def get(self, identifier: str):
|
|
try:
|
|
with db_readonly() as conn:
|
|
shared_repo = SharedConversationsRepository(conn)
|
|
conv_repo = ConversationsRepository(conn)
|
|
attach_repo = AttachmentsRepository(conn)
|
|
|
|
shared = shared_repo.find_by_uuid(identifier)
|
|
if not shared or not shared.get("conversation_id"):
|
|
return make_response(
|
|
jsonify(
|
|
{
|
|
"success": False,
|
|
"error": "might have broken url or the conversation does not exist",
|
|
}
|
|
),
|
|
404,
|
|
)
|
|
conv_pg_id = str(shared["conversation_id"])
|
|
owner_user = shared.get("user_id")
|
|
|
|
conversation = conv_repo.get_owned(conv_pg_id, owner_user) if owner_user else None
|
|
if conversation is None:
|
|
# Fall back to any-user lookup in case shared row's
|
|
# user_id is missing — still keyed by PG UUID.
|
|
row = conn.execute(
|
|
_sql_text(
|
|
"SELECT * FROM conversations WHERE id = CAST(:id AS uuid)"
|
|
),
|
|
{"id": conv_pg_id},
|
|
).fetchone()
|
|
if row is None:
|
|
return make_response(
|
|
jsonify(
|
|
{
|
|
"success": False,
|
|
"error": "might have broken url or the conversation does not exist",
|
|
}
|
|
),
|
|
404,
|
|
)
|
|
conversation = dict(row._mapping)
|
|
|
|
messages = conv_repo.get_messages(conv_pg_id)
|
|
first_n = shared.get("first_n_queries") or 0
|
|
conversation_queries = []
|
|
for msg in messages[:first_n]:
|
|
query = {
|
|
"prompt": msg.get("prompt"),
|
|
"response": msg.get("response"),
|
|
"thought": msg.get("thought"),
|
|
"sources": msg.get("sources") or [],
|
|
"tool_calls": msg.get("tool_calls") or [],
|
|
"timestamp": (
|
|
msg["timestamp"].isoformat()
|
|
if hasattr(msg.get("timestamp"), "isoformat")
|
|
else msg.get("timestamp")
|
|
),
|
|
"feedback": msg.get("feedback"),
|
|
}
|
|
attachments = msg.get("attachments") or []
|
|
if attachments:
|
|
attachment_details = []
|
|
for attachment_id in attachments:
|
|
try:
|
|
attachment = attach_repo.get_any(
|
|
str(attachment_id), owner_user,
|
|
) if owner_user else None
|
|
if attachment:
|
|
attachment_details.append(
|
|
{
|
|
"id": str(attachment["id"]),
|
|
"fileName": attachment.get(
|
|
"filename", "Unknown file"
|
|
),
|
|
}
|
|
)
|
|
except Exception as e:
|
|
current_app.logger.error(
|
|
f"Error retrieving attachment {attachment_id}: {e}",
|
|
exc_info=True,
|
|
)
|
|
query["attachments"] = attachment_details
|
|
conversation_queries.append(query)
|
|
|
|
created = conversation.get("created_at") or conversation.get("date")
|
|
date_iso = (
|
|
created.isoformat()
|
|
if hasattr(created, "isoformat")
|
|
else (str(created) if created is not None else None)
|
|
)
|
|
res = {
|
|
"success": True,
|
|
"queries": conversation_queries,
|
|
"title": conversation.get("name"),
|
|
"timestamp": date_iso,
|
|
}
|
|
if shared.get("is_promptable") and shared.get("api_key"):
|
|
res["api_key"] = shared["api_key"]
|
|
return make_response(jsonify(res), 200)
|
|
except Exception as err:
|
|
current_app.logger.error(
|
|
f"Error getting shared conversation: {err}", exc_info=True
|
|
)
|
|
return make_response(jsonify({"success": False}), 400)
|