A hyperparameter is a setting that decides how an AI learns, chosen before training by people (or by a tool that searches for settings). Here is how it differs from a parameter, what happens when you change the learning rate, batch size and number of epochs, grid search versus random search, why you choose on a validation set, and how big AIs are tuned, explained with pictures and a quick quiz.

The Julia Programming Language

There are comments on PubPeer for publication: Hybrid CNN-LSTM model with efficient hyperparameter tuning for prediction of Parkinson's disease (2023)

Modern optimization and scaling laws for k-means clustering

Large-scale ranking and recommendation models of Web platforms benefit from increased compute much as large language models do, and their scaling behavior is broadly power-law in form. Yet ranking mod...

Use successive halving in scikit-learn to allocate resources progressively and find strong configurations more efficient

Master hyperparameter tuning for linear models, random forests, and gradient-boosting trees with grid & random search, Bayesian optimization, and multi-fidelity methods.

Modelers.cn — Huawei's Ascend-focused AI model hub — has a new small reasoning model worth a look if you like your experiments transparent.…

Your model's most dangerous vulnerability is rarely a sub-optimal hyperparameter configuration. It is hidden outliers lurking undetected inside your training and evaluation data. In production, vision models frequently fail on rare scenarios that were never surfaced during development: unusual sensor artifacts, unrepresented object variants, corrupted metadata, or bizarre edge-case interactions. Traditional statistical checks—like tracking image resolution, file size, or mean pixel values—fail to detect semantic and compositional anomalies. By leveraging foundation model embeddings, density-based anomaly algorithms like Local Outlier Factor (LOF), and dimensionality reduction with UMAP, you can systematically reveal hidden dataset structure and isolate rogue outliers before training begins. EMBEDDING-BASED OUTLIER DETECTION PIPELINE +-------------------------------------------------------------+ | Raw Image Dataset | | | | | v | | Foundation Model Embedding (CLIP / DINOv2 / SigLIP) | | | | | +-----------------------------+ | | | | | | v v | | Local Outlier Factor (LOF) UMAP 2D Projection | | [Quantitative Anomaly Score] [Visual Cluster Exploration] | | | | | | +-----------------------------+ | | v | | Visual Inspection & Triage in FiftyOne | +-------------------------------------------------------------+ ## The Three Classes of Visual Outliers When inspecting anomalies in visual datasets, outliers typically fall into three distinct technical categories: 1. **Quality and Sensor Anomalies:** Corrupted frames, extreme lens flare, severe compression blockiness, dropped video packets, or infrared sensor bleed. These represent data pipeline defects that should almost always be scrubbed or quarantined. 2. **Semantic and Content Anomalies:** Rare, valid physical scenarios that occur with low frequency: an overturned truck on a highway, an animal running across an industrial floor, or a worker wearing non-standard personal protective equipment. These are valuable operational edge cases that must be preserved and reinforced. 3. **Annotation and Formatting Bugs:** An image tagged with a label from an entirely different ontology, bounding boxes scaled incorrectly, or labels assigned to inverted coordinate axes. Identifying which category an outlier belongs to determines whether you prune the sample, fix its metadata, or curate more examples to support it. ## Extracting Dense Semantic Embeddings Pixel-level distances cannot distinguish between a dark cat and a black leather jacket. Foundation vision models trained via self-supervised learning (DINOv2) or contrastive language-image pretraining (CLIP, SigLIP) solve this by encoding high-level semantic, structural, and stylistic properties into dense feature vectors. import fiftyone as fo import fiftyone.brain as fob # Load dataset dataset = fo.load_dataset("industrial_inspection_v2") # Compute embeddings using a foundation vision backbone fob.compute_similarity( dataset, model="clip-vit-base32-torch", embeddings_field="clip_embeddings", brain_key="clip_sim" ) print("Computed dense embeddings across all dataset samples.") ## Density-Based Anomaly Scoring with Local Outlier Factor A common mistake in outlier detection is calculating global Euclidean or Mahalanobis distance from the dataset centroid. Visual datasets are inherently multimodal: images of outdoor scenes form one dense cluster, while indoor scenes form another with completely different local density. Local Outlier Factor (LOF) solves this by comparing the local density of an image's embedding to the density of its $k$-nearest neighbors: $$\text{LOF}_k(p) = \frac{\sum_{o \in N_k(p)} \frac{\text{lrd}(o)}{\text{lrd}(p)}}{|N_k(p)|}$$ Where $\text{lrd}(p)$ is the local reachability density of point $p$. An LOF score substantially greater than $1.0$ indicates that the sample resides in a sparse region relative to its neighbors, flagging it as an anomaly. from sklearn.neighbors import LocalOutlierFactor import numpy as np # Extract embeddings array from FiftyOne dataset embeddings = np.array(dataset.values("clip_embeddings")) # Fit Local Outlier Factor lof = LocalOutlierFactor(n_neighbors=20, contamination=0.02) outlier_labels = lof.fit_predict(embeddings) anomaly_scores = -lof.negative_outlier_factor_ # Store anomaly scores back onto the dataset dataset.set_values("outlier_score", anomaly_scores.tolist()) # Query the top 1% most severe anomalies outliers_view = dataset.sort_by("outlier_score", reverse=True).limit(50) print(f"Isolated top {len(outliers_view)} outlier candidates for inspection.") ## Visualizing Dataset Manifolds with UMAP Quantitative anomaly scores tell you _which_ images are unusual, but dimensionality reduction tells you _why_. Uniform Manifold Approximation and Projection (UMAP) preserves both local and global neighborhood structures when projecting high-dimensional embeddings down to a 2D plane: # Compute 2D UMAP visualization coordinates fob.compute_visualization( dataset, embeddings="clip_embeddings", method="umap", brain_key="umap_2d" ) # Launch FiftyOne interactive app to inspect clusters session = fo.launch_app(dataset) In the interactive 2D scatter plot: * **Tight, dense clusters** represent standard, well-represented operational scenarios. * **Isolated satellites and rogue points** floating outside major clusters pinpoint rare outliers, mislabeled samples, and corrupted images. Outlier Type | UMAP Geometry | Typical Root Cause | Action Plan ---|---|---|--- **Pipeline Defect** | Far off the manifold, isolated single point | Corrupted sensor frame, black frame | Quarantine and remove from training set **Annotation Error** | Sits inside Cluster A, but labeled as Class B | Human labeler typo or ambiguous definition | Correct ground truth annotation **Operational Edge Case** | Small cluster of 3–5 points detached from main body | Rare physical scenario in the real world | Preserve, balance, and collect companion cases ## Turning Anomaly Detection into Practice Running outlier detection during data ingestion prevents corrupted frames from skewing model weights and ensures that critical real-world edge cases receive explicit attention. To explore related evaluation workflows, review our guide on curating golden trajectories for red-teaming and analyzing vision datasets. To get hands-on experience running embedding searches, LOF audits, and UMAP clustering on real datasets, audit Harpreet Sahota's Coursera course: Hands-on Data Centric Visual AI. ## Frequently asked questions ### What makes visual outliers dangerous in machine learning datasets? Visual outliers can corrupt model decision boundaries, cause catastrophic gradients during training, or represent critical real-world edge cases that models fail to recognize in production. Detecting them early prevents both training instability and operational blind spots. ### Why are foundation model embeddings better than pixel metrics for outlier detection? Raw pixel comparisons only capture low-level color and brightness shifts. Foundation model embeddings (such as CLIP, DINOv2, or SigLIP) map images into a structured semantic space where distance reflects conceptual, compositional, and contextual similarity. ### How does Local Outlier Factor (LOF) differ from simple Euclidean distance? Global Euclidean distance assumes uniform density across the dataset. Local Outlier Factor compares the local density of an object to the local densities of its k-nearest neighbors, correctly identifying anomalies even in regions with variable cluster densities. ### Where can developers practice embedding-based curation workflows? Complete hands-on code examples and guided dataset curation labs are available in Harpreet Sahota's Coursera course, Hands-on Data Centric Visual AI. _This post first appeared as the original article. I'm Harpreet Sahota, Hacker-in-Residence at The Employable Data Scientist, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the Artists of Intelligence podcast. For new field notes by email, subscribe on Substack._

Build framework-agnostic optimization workflows with advanced samplers, pruning, and automated model selection.

The rubber hits the road on ColabFold! I hope that's rubber I'm smelling — although it may be the computer. A replication attempt of a protein fold figure in K. Pedram et al (2023).

Abstract page for arXiv paper 2610.37842: Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling

ArXiv link for Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling