MODA_NER
An open benchmark and model family for fashion attribute extraction. Four tracks, never averaged. Predictions are hashed before labels are opened, scorers fail closed, and the runs we lose are published next to the ones we win.
The claim, in full
MODA General is the best of the named open systems evaluated under the frozen public MODA General Attribute Suite, spanning localized garment crops, catalog product images, color and fit, and applicability-aware full-body attributes.
Not world-best, not universal state of the art, not human-gold quality, not production readiness. The claim was frozen over the three image tracks; the text track was added afterwards and is deliberately outside it, because no external system has yet been evaluated on it under identical conditions.
Four tracks
No single public dataset covers fashion attributes at the breadth a catalogue needs, and forcing several into one label space hides exactly the trade-off that matters. So the tracks stay separate and the scores are never averaged.
| Track | Input | Frozen test | Resampled by | Colour | Fit | "Not applicable" |
|---|---|---|---|---|---|---|
crop | cropped garment | 4,688 crops / 1,158 images | source image | no | no | partial |
catalog | product image | 9,995 images / 61,384 cells | image | yes | yes | no |
fullbody | full-body photo | 5,000 images / 1,751 groups | product group | no | no | yes, explicit |
text | title or description | 1,071 rows | row | yes | yes | n/a |
Colour and fit, the two attributes merchandisers ask about first, exist in only two of the four. Exactly one attribute is present in all four: neckline. It is also the field that has cost us the most, because two sources can divide it differently and nothing warns you.
Results
| Track | What the score measures | MODA | Comparator | 95% interval | Recompute |
|---|---|---|---|---|---|
crop | share of predictions matching exactly, 15 fields | 0.6300 | 0.6245 | [+0.0014, +0.0097] | yes |
catalog | score per field, averaged over 10 fields | 0.8292 | 0.6657 | [+0.1595, +0.1676] | yes |
fullbody overall | score per field, averaged over 18 fields | 0.6917 | 0.5943 | [+0.0891, +0.1053] | yes |
fullbody knows when not to answer | reliability of saying an attribute is absent | 0.6637 | 0.6088 | [+0.0433, +0.0657] | yes |
fullbody when visible | the 18-field average on present attributes | 0.5785 | 0.4969 | [+0.0723, +0.0905] | yes |
Do not read across rows. Different images, different fields, different metrics. Ten catalogue fields on clean studio photos is an easier problem than fifteen on a cropped garment; 0.8292 is not "better" than 0.6300. Each row compares one system against one comparator on one track and says nothing else.
Comparators: on catalog and fullbody, FashionCLIP 2.0 with matched supervised heads, a genuine external system. On crop, the 0.6245 is our own architecture on a frozen third-party encoder, so that row is an encoder ablation rather than a win over another vendor. The real third-party baselines there are zero-shot and score 0.1805 and 0.1817, on a task they were not built for, which is why we do not lean on the gap.
Per field, on crop
The headline pools every attribute decision. It hides a spread of more than fifty points, and the spread is what determines whether this is usable in your pipeline.
| Field | F1 | Labelled cases | Field | F1 | Labelled cases |
|---|---|---|---|---|---|
master_category | 0.9215 | 4,688 | sub_category | 0.5966 | 1,594 |
category | 0.8825 | 4,688 | silhouette | 0.5535 | 1,994 |
pattern | 0.8356 | 1,994 | waist_type | 0.5002 | 1,102 |
sleeve_length | 0.8073 | 912 | neckline | 0.4650 | 1,126 |
closure_type | 0.7545 | 1,080 | collar_style | 0.4566 | 287 |
collar_presence | 0.7508 | 294 | surface_treatment | 0.4398 | 880 |
sleeve_shape | 0.6787 | 855 | material | 0.4148 | 146 |
hemline | 0.6428 | 1,729 | |||
Taking F1 0.75 as a working line, six of fifteen fields are safe to populate automatically and nine need a review step. On 49.66% of garments every one of the fifteen fields is right. Telling a coat from a dress is close to solved; telling denim from twill is not.
Models
| Model | Input | Weights | Commercial use |
|---|---|---|---|
MODA_NER(V) Crop moda-ner-v-crop | cropped garment | MIT | permitted |
MODA_NER(V) Catalog moda-ner-v-catalog | product image | CC BY-NC 4.0 | non-commercial |
MODA_NER(V) Full-body moda-ner-v-fullbody | full-body photo | CC BY-NC 4.0 | non-commercial |
| MODA_NER(T) | product text | not distributed | — |
| MODA_NER Pro | declared schema | hosted | talk to us |
Two routes are evaluated against research-only corpora whose terms extend to derived data, so their weights are non-commercial. That restriction binds us as well: those exact weights are not part of Hopit's paid product. We are not holding back a better model either — the published crop checkpoint is the same one that produced 0.6300.
Reproduce
We do not redistribute source datasets or their labels. You obtain each corpus from its original source under that source's terms, and the builders reconstruct the frozen split from record IDs and checksums. What ships here is the protocol, the scorers, the uncertainty code, and our own prediction files with their hashes.
python -m suite.crop.score \ --gold <your rebuilt split> \ --predictions results/crop/moda-ner-v-crop/evaluation_predictions.jsonl \ --output /tmp/recomputed.json
That regenerates 0.6300 exactly. Every published figure on this page has been recomputed from its shipped prediction file and matches, including all three fullbody confidence intervals from the 10,000-sample bootstrap clustered over 1,751 product groups.
Scoring your own model
Freeze your checkpoint. Predict on the same record IDs without reading labels. Commit the prediction hash. Score each track and report the paired interval against our published predictions. If you beat us under this protocol we would rather hear it than not.
How the harness stays honest
| Rule | What it prevents |
|---|---|
| Freeze the test before the model runs | the target moving while you measure |
| Split and resample at the natural unit | near-duplicates inflating a score, and intervals that are too narrow |
| Hash predictions before labels open | substituting a better file after seeing the answers |
| Fail closed on missing or unknown rows | quietly dropping the cases a model got wrong |
| Score "not visible" separately from "wrong" | rewarding a model that invents an attribute for every garment |
| Require a positive interval on every track | a weak track hiding inside an average |
What this does not tell you
Every label here is dataset-native, not independent human gold; an independent annotation programme is committed and unfinished. A benchmark cannot prove that its taxonomy matches yours — on one external set a neckline vocabulary mismatch cost a production route fourteen points with no model regression at all. And a strong extraction score does not transfer to adjacent tasks: on mapping titles to a retailer's own product taxonomy, a terminal n-gram baseline beats our text model.