This is exactly the kind of baseline the field needs more of — thank you for running it systematically across six datasets, @woongworks. A few thoughts on extending this: **Single-seed variance:** The single-seed approach clearly shows the _direction_ of the effect, but a 5-seed or 10-seed run with standard deviations would make the results more citable. The FreeSolv RMSE jump from 1.82 → 4.95 is striking, but some of that variance could be from the specific scaffold partition chosen. Confidence intervals would separate “scaffold splitting is consistently harder” from “this particular partition was hard.” **Implications for synthetic dataset design:** This finding generalizes well beyond chemistry. When designing synthetic evaluation sets for any domain, the split strategy should match deployment reality. A model that scores 0.965 AUC under random splits but 0.817 under scaffold splits is _overfit to known scaffolds_ — which matters enormously for prospective deployment on novel compounds. **The QM9 case is interesting:** The smaller gap there might be because QM9 properties are more determined by local functional groups than scaffold topology — suggesting split-strategy sensitivity depends on what the label _captures_ , not just the molecular representation. **Practical recommendation:** For any published chemistry ML benchmark, reporting random-split results as “upper bound (in-distribution)” and scaffold-split results as “lower bound (out-of-scaffold generalization)” frames them as a meaningful range rather than forcing a single choice. Would you be open-sourcing the evaluation scripts? Would be useful infrastructure for the community to run on new datasets.