DIY: Research Data Platform
A market-research data platform for creative testing: survey authoring, cohort management, a metrics dashboard and spreadsheet export. Four NestJS services on Prisma over a 190-model MySQL schema, alongside two older Express generations and an ETL that keeps a MongoDB analytics store in step with the relational system of record.
Overview
A client configures a study, selects the content to be tested, fields it to a cohort, and gets results back as a dashboard and a spreadsheet. It is the largest system I have worked on, and it is a data platform rather than an application: surveys, cohorts, content testing, telemetry ingestion and reporting each have their own service, and the schema behind them runs to 190 models.
The current generation
A NestJS monorepo of four apps — dashboard, respondent-facing, survey creation and data transfer — over shared libraries for Prisma, S3, RabbitMQ and utilities. Each app validates its own environment with Joi at boot. The Prisma layer runs the MariaDB driver adapter with tuned connect and acquire timeouts, a soft-delete extension, and a query hook that logs only mutating statements to an audit log — reads are noise, writes are evidence.
Two databases, on purpose
MySQL is the system of record; MongoDB is the analytics store. A transfer service exposes the relational survey graph over HTTP and a migration controller on the Mongo side pulls it across in bulk, parsing JSON-string and comma-separated columns back into real types on the way and diffing an identity pair against what already exists so a re-run is a no-op. There is an inverse operation too, which matters more than it sounds: a migration you cannot undo is one nobody dares run twice.
Three problems worth naming
Survey questions form a graph, not a list — a question can hang off an option, a grid or a numeric — so resolving a survey means walking parent-to-child edges iteratively with a cycle guard rather than recursing and hoping. Uploaded media is probed before it is touched: only files outside the accepted codecs or above a bitrate threshold get transcoded, and the transcodes are gated to one at a time so a batch upload cannot saturate the box. And exactly one query in the codebase is raw SQL — an eleven-table join that collapses options, grids, numerics and free text into a single row set. Knowing when to stop using the query builder is part of using it well.
Deployment
AWS CodeDeploy with lifecycle hooks and PM2, pulling configuration from SSM Parameter Store at install time rather than baking an environment file into the artefact — the deployment carries no secrets, and rotating one does not require a redeploy.
Honest scope
This one is a team system and I am not its main author. On the current backend I am third of four contributors; on the older Express and Mongo generation I am second of eight; the internal insights front end and the SQL port of the content-testing API are mine outright. What I can point at is the transfer and reporting side and the work of keeping two data models honest with each other. The measurement methodology at the heart of the product is my employer's and is not described here.