Systems Architect - Data Integration & AI Pipeline Design

Job DescriptionJob Description

You will design the end-to-end architecture for a peptide protocol platform that pulls structured and unstructured data from Reddit, PubMed, and YouTube, then turns it into trustworthy, searchable knowledge. Your work covers the blueprinting phase in full: data models in PostgreSQL, normalized schemas for source attribution, API contracts, and the orchestration layer that coordinates ingestion, enrichment, and indexing. You will define integration patterns for PRAW, Entrez, and YouTube Data API, plan rate-limit aware retries and backoff, and map failure paths so the system heals itself without data loss.

You will design a creator claim and verification flow so Reddit users can prove ownership and attach their contributions. That includes OAuth for Reddit, state machines for claim review, dispute handling, and auditable attribution that persists across updates and merges. You will specify deduplication logic that reconciles overlapping protocols from multiple sources while preserving provenance.

You will choose and justify the core stack: Python services with Celery or Airflow for orchestration, Redis for caching, RabbitMQ or SQS for job queues, Elasticsearch for search, and OpenAI or Claude for NLP tasks like protocol extraction and entity linking. You will produce clear system diagrams, sequence charts, and written specs that an implementation team can follow without guesswork. You will review work products for alignment and advise on scaling from prototype to production.

This role is architecture first, not hands-on coding. You will not own frontend work or day-to-day bug triage. You will collaborate on priorities, milestones, and an implementation roadmap, and you will help the team make tradeoffs that keep data quality, attribution integrity, and system reliability front and center.