Reference · v1

Generative Image Evaluation Metrics

Working reference for evaluating text-to-image and avatar generation output — standard metrics, VLM-as-judge approaches, and per-module methods for face, outfit, and world.

Standard Metrics

CLIP Score

Measures text-to-image alignment. Using OpenAI's CLIP (Contrastive Language-Image Pre-training) model, it scores how accurately the visual content of a generated image matches the literal meaning of the prompt text.

Fréchet Inception Distance (FID)

The gold standard for assessing image synthesis. It compares a large batch of generated images against a batch of real images. By extracting high-level features, it measures how realistic and diverse the outputs are. A lower FID score indicates closer alignment with reality.

Inception Score (IS)

An older but still utilized metric that evaluates two things: whether an image contains a clear, recognizable object (low entropy per image) and whether the model can generate a wide variety of different objects (high total entropy across the batch).

Visual Question Answering (VQA)

Automated evaluation scripts ask a separate VLM precise questions about a generated image (e.g., "Is there a red apple on the table?" or "Count the number of fingers on the left hand"). If the VLM answers correctly based on the original prompt, the image generator scores high on instruction following.

https://dl.acm.org/doi/10.1145/3728635?__cf_chl_f_tk=tk2afAhUDvjuGqySdvpxqTvdGwEl3SKctRguURQsVAc-1783409710-1.0.1.1-T3lPaMgtNBrb5sRiriOrkXqbNZKCWr0PoolW_UOQzcI#sec-1

Proprietary Quality Networks

Frameworks like the QMI-Net (Question-guided Multimodal Interaction Network) combine visual features with VLM text prompts to directly score aesthetic quality and text-consistency simultaneously.

Question-Guided Multimodal Interaction (QMI / VQA)

Instead of outputting a generic score, the network runs a highly optimized loop of visual questions and answers. The network automatically asks itself targeted questions based on the input prompt:

  • Question: "Are there exactly three cars in the background?"
  • Network Action: It routes focus to the background vectors, isolates the elements, counts them, and generates an algorithmic probability.

Label-Free Relative Supervision (The ELIQ Framework)

Historically, training an AI judge required thousands of humans to manually rate images with a Mean Opinion Score (MOS). New enterprise Quality Networks utilize a Label-Free approach (like the ELIQ framework). They use an automated instruction-tuning process to convert a frozen model backbone into an aspect-aware quality evaluator, using self-generated relative rankings rather than static human scores.

No-Reference Image Quality Assessment (NR-IQA)

Quality Networks use No-Reference Transformers. They look at both local features (like individual pixel blurring) and non-local features (global composition) to predict perceptual quality entirely in a vacuum.

Andrej Karpathy's Grid Methodology

Instead of grading individual photos one by one, he suggests outputting options as a unified grid inside a single canvas. This allows the evaluator to compare variations in layout, density, and tone side-by-side to clearly map out trade-offs.

Other Specifics

Module 01

Face Module: Biometric & Facial Feature Verification

When your API creates an avatar face, it must maintain character identity, proportions, and expression realism.

ArcFace / FaceNet Identity Similarity
  • How it works: You pass your generated avatar image through a highly localized facial recognition network like ArcFace or FaceNet to extract a 512-dimensional facial embedding vector.
  • What it catches: It calculates the cosine similarity against a target face or previous generations. If your avatar API is supposed to generate the "same character in different worlds," this checks if the face structurally morphs into a different person.
3D Morphable Models (3DMM) & Landmark Fit
  • How it works: Tools extract 68 or 468 precise 3D facial landmarks from the 2D image output.
  • What it catches: It calculates Landmark Loss and Contour Edge Loss. It programmatically flags if the avatar's eyes are misaligned, if the jaw structure is warped, or if facial expressions look uncanny or broken.
Face Parsing Networks (Skin Mask & Geometry)
  • How it works: Segments the face into discrete labels (lips, nose, eyes, skin).
  • What it catches: Measures Photometric Loss across skin tone distributions. It ensures that lighting shifts in the "World" don't randomly change the actual structural ethnicity or skin color mapping of the character.
Module 02

Outfit Module: Segment Geometry & Style Consistency

Evaluating the clothing layers involves checking for graphic fidelity, prompt alignment, and clipping artifacts.

Layered Semantic Segmentation (SAM & ClothesNet)
  • How it works: An object segmentation tool like Meta's Segment Anything Model (SAM) isolates the clothing item coordinates exclusively.
  • What it catches: It checks the exact pixel mask boundaries. It flags layer overlapping errors—such as a jacket texture accidentally clipping into the avatar's skin or pants melting into the background texture.
Gram Matrix Style Transfer Metrics (ArtFID)
  • How it works: Instead of reading text, Gram Loss isolates the pure artistic texture, weave pattern, and color distribution of the garment.
  • What it catches: If a prompt asks for a "leather jacket" or a "silk dress," Gram Loss measures if the reflective shaders match the material properties of leather or silk, rather than just random flat colors.
Module 03

World Module: Depth, Perspective & Coherence

The background environment or "world" needs to be grounded realistically around the character.

Depth Map & Monocular Depth Consistency (MiDaS)
  • How it works: Pass the generated scene through a depth estimation model like MiDaS to create a grayscale Z-axis depth gradient map.
  • What it catches: It checks if the avatar's feet are mathematically resting on the physical ground. If a depth map shows the avatar has the exact same depth score as a distant mountain, it catches a perspective blending error.
Natural Scene Statistics (NSS) / NIQE Scores
  • How it works: The Natural Image Quality Evaluator (NIQE) measures how much the background environment deviates from the mathematical distributions of real, physical photography landscape configurations.
  • What it catches: It spots AI hallucinations in the background—like a floating window, distorted building horizons, or chaotic cloud formations that ruin immersion.

Side reads