Ingestion flow
Every source runs the same template
fetchnormalizeresolve entity_idPostgresMongoNeo4jembed → Mongo vectorsSSE + metrics
A canonical entity_id links the same record across Postgres, Mongo and Neo4j. After each run an event is published to Redis → SSE (live UI refresh) and any subscribed living metric recomputes.
PostgreSQL
identity · structured · vectors
MongoDB
document lake
Neo4j
graph · 0 edges
Full-text scraper
scrape → key highlights → embed, across news articles
Pipelines
Ingestion sources
Scheduled jobs
Celery Beat
| Job | Schedule (cron) |
|---|
Agentic workflow runs
DAG executions