Data pipeline diagram

Diagram

Draw.io source: documentations/architecture/JobSim_Simple_Diagrams.drawio

Stage	Writes	Reads	Source
API write	`messages`, `conversations`, `channels.updated_at`, `simulation_state_events`, `unread_counts`	auth context, channel/conversation membership, simulation lifecycle	`src/core/messages/message_create_service.py`
Agent processing	agent replies in `messages`; diagnostics in `autogen-logs`, `simulation_audit_events`, `ai_usage_events`	messages, characters, settings, tasks, memory, workplace context	`src/core/messages/message_background.py`, `src/core/agents/agent_monitor.py`
Agent memory	`agent_memories`, `agent_user_memory_summary`; optional mem0 store	scoped message history, org policy, org docs	`src/core/autogen/mongo_driven_memory.py`, `src/core/autogen/mem0_memory.py`
Knowledge indexing	Qdrant vectors in `jobsim_org_knowledge`	`org_knowledge_docs`	`src/memory/org_knowledge_qdrant.py`
Analytics	`communication_analysis`, `user_metrics`, `ai_usage_events`	messages, users, existing metrics	`src/core/analytics/analytics_processor.py`, `src/core/analytics/user_analytics.py`
Task evaluation	`task_evaluations`, follow-ups, consequences, relationship edges, competency snapshots, session assessments	submissions, task criteria, org knowledge, workplace frame	`src/core/tasks/task_evaluation_service.py`, `src/core/tasks/task_followup_persistence_service.py`
Replay and diagnostics	read-only response shaping	`autogen-logs`, `simulation_audit_events`, state events, outcomes	`src/apis/simulation_admin/replay.py`, `src/apis/simulation_admin/analytics.py`

MongoDB is source of truth for product state: users, orgs, channels, conversations, messages, tasks, settings, characters, evaluations, audit state, and outcomes.
Qdrant stores derived vectors only. Payloads include organization_id, doc_id, title, chunk index, and chunk text. Retrieval enforces org scope in Qdrant filters.
mem0 is optional long-horizon memory and is attached per agent only when org settings enable it.
autogen-logs has 30-day TTL. simulation_audit_events has 90-day TTL. Both are debug/audit streams, not primary domain state.
ai_usage_events is normalized cost telemetry. Legacy token_usage can be backfilled into it.