Day 44: Fronting with NestJS; measuring baseline latency
Wiring it into the stack, and the number that matters
NestJS fronts the Python classifier exactly as it will front every inference service through Stage 6 — the gateway/UI stays in your stack, Python does inference behind it. Today's real deliverable, though, is a *measurement*: the classifier's inference latency on the CPU droplet. The roadmap is explicit that this number is your Stage 6A 'before' baseline. Optimization without a baseline is theater; write it down.
@Controller('garments')
export class GarmentController {
@Post('classify')
@UseInterceptors(FileInterceptor('file'))
async classify(@UploadedFile() file: Express.Multer.File) {
const form = new FormData();
form.append('file', new Blob([file.buffer]), file.originalname);
const started = performance.now();
const res = await fetch(`${process.env.CLASSIFIER_URL}/classify`, {
method: 'POST', body: form,
});
const latencyMs = performance.now() - started;
const result = await res.json();
return { ...result, latencyMs }; // surface latency so it gets logged
}
}Measure honestly
- Report percentiles (p50, p95, p99), not just an average — tail latency is what users feel (a lesson that returns in force in Stage 5).
- Measure the model inference time and the end-to-end time separately, so you know how much is the model vs the network/serialization overhead.
- Warm up first — the first inference after load is slower; measure steady state.
The baseline is a promise to your future self
In Stage 6A you'll take this model through torch.compile → ONNX → TensorRT → INT8 and claim a speedup. That claim is only credible against a recorded starting point. 'The classifier does ~180ms p95 on CPU, single-threaded, batch 1, warm' — that specificity is what makes the later '4× faster' defensible in an interview.
Key terms
- Baseline latency
- The recorded inference time of the unoptimized model, the reference point all later optimization is measured against.
- Percentile latency (p50/p95/p99)
- The latency below which that percentage of requests fall; captures tail behavior an average hides.
- Warm-up
- Running a few inferences before measuring, to exclude one-time initialization costs from steady-state numbers.
Why does the roadmap insist you record the classifier's baseline latency now, in Stage 1?
