What Is a Generative Model in Machine Learning?

A generative model learns the underlying distribution of data so it can produce new samples that look like the training data. Discriminative models learn boundaries between classes instead. This piece explains the maths in plain language, walks through the main families (Naive Bayes, VAEs, GANs, diffusion, autoregressive transformers), and shows where the idea actually shows up in work.

The definition, without the hand-waving

A generative model in machine learning is a model that learns the probability distribution of the data itself, so it can produce new samples that plausibly could have come from the same source. Formally, it models the joint probability P(x, y), or in the unsupervised case just P(x), where x is the data and y is the label. A discriminative model does something narrower: it learns P(y | x), the probability of a label given the data, and nothing else.

That difference sounds academic. It isn't. A discriminative spam classifier learns a boundary that separates spam from not-spam. It can tell you which side an email falls on. Ask it to write you a spam email and it has no idea, because it never learned what spam looks like, only where the line sits. A generative model of spam learns what spam emails are made of, and as a side effect it can both classify and produce them.

Here is the analogy I use with students. A discriminative model is a border guard who has memorised which passports get through. A generative model is a forger who has studied enough passports to make a new one. The forger can also work as a border guard. The guard can never work as a forger.

Why learning P(x) is so much harder

Modelling a boundary is cheap. Modelling a distribution is brutal.

Consider a 256x256 colour image. That is 196,608 numbers, each with 256 possible values. The space of all possible images of that size is astronomically large, and the tiny sliver of it that looks like a human face is a vanishingly thin surface inside that space. A generative model has to learn the shape of that surface well enough to walk along it and land somewhere new that still looks like a face. A classifier only has to decide, for a point already handed to it, whether it is on the cat side or the dog side.

This is why generative modelling lagged classification by years. ImageNet classification was largely solved around 2015. Genuinely convincing image generation arrived closer to 2021 and 2022, with diffusion models. The gap was not compute alone. It was that nobody had a training objective that made high-dimensional density modelling stable, and several that people tried (GANs especially) were notoriously temperamental.

The main families, and what each one actually does

There is no single algorithm called "the generative model". It is a category, and the members work on very different principles. Older ones like Naive Bayes are generative in the textbook sense and were being used for spam filtering in the 1990s. The recent ones are what people mean when they say generative AI.

Autoregressive models deserve a note, because they are the family behind ChatGPT, Claude and Gemini. They factor the joint probability into a chain: the probability of the whole sequence is the product of the probability of each token given everything before it. Train on enough text and you get a model that can continue any prefix. That is all next-token prediction is, and it turns out to be enough to produce something that looks like reasoning.

Family	Core idea	Typical use	Trade-off
Naive Bayes	Models P(x|y) per class with a strong independence assumption	Text classification, spam	Fast and interpretable, assumptions rarely hold
Gaussian Mixture Models	Data as a blend of Gaussian clusters	Clustering, anomaly detection, speech	Breaks down in high dimensions
Variational Autoencoders (VAEs)	Compress to a latent space, sample from it, decode	Representation learning, molecule design	Stable to train, outputs often blurry
GANs	A generator and a discriminator competing	Fast image synthesis, super-resolution	Sharp results, unstable training, mode collapse
Diffusion models	Add noise step by step, learn to reverse it	Image, video and audio generation	Very high quality, slow at inference
Autoregressive transformers	Predict the next token given all previous ones	LLMs, code, music	Excellent at sequences, expensive to train

Where this distinction actually bites in practice

If you are building something, the generative-versus-discriminative question is a real design decision, not trivia.

Anomaly detection is the clearest case. You often have thousands of normal transactions and almost no fraud examples. A discriminative classifier needs both classes to learn a boundary and will fail with that imbalance. A generative model of normal behaviour just needs the normal data. Anything it assigns low probability to is suspicious. Fraud teams at banks have used this logic for years, long before it was fashionable.

Data scarcity is another. Generative models can produce synthetic training data, which is standard practice in medical imaging where real labelled scans are limited and privacy-restricted. But there is a catch worth knowing: train generative models repeatedly on their own outputs and quality degrades, a failure mode researchers call model collapse.

And then there is the honest downside. Generative models hallucinate because generating is what they do. An LLM producing a confident wrong citation is not malfunctioning. It is sampling from a distribution that assigns non-zero probability to plausible-sounding nonsense. Once you internalise that a generative model outputs likely things rather than true things, most of the mystery around LLM failures disappears.

What to do with this if you are learning ML

Reading definitions gets you to the point where you can answer an interview question. It does not get you to the point where you can build. The jump happens when you implement something small and watch it fail.

A reasonable sequence, in order of difficulty:

• Fit a Gaussian Mixture Model on a 2D toy dataset and sample from it. Twenty lines of scikit-learn. You will see density modelling with your own eyes.
• Train a VAE on MNIST. Look at the latent space. Interpolate between two digits and watch one morph into the other.
• Implement a tiny GAN and let it mode-collapse on you. The frustration teaches more than the paper does.
• Write a character-level autoregressive model from scratch, then read Karpathy's nanoGPT and see what you got wrong.
• Only then touch diffusion. It makes far more sense once you already think in terms of distributions.

The part most explanations skip

Most people searching this question are not planning to train a generative model from scratch. They want to understand what is happening inside the tools they already use at work, and to stop nodding along in meetings where someone says "we should fine-tune a generative model for this".

For that, the useful mental shift is small. Stop thinking of these systems as answer machines and start thinking of them as samplers. When you prompt an LLM, you are not querying a database. You are conditioning a probability distribution and drawing from it. That single reframe explains temperature settings, why the same prompt gives different outputs, why retrieval augmentation helps, and why prompt wording changes results so much.

If you want to go from understanding the concept to actually building with it, that gap is usually a matter of guided practice rather than more reading, which is what the AI Fellowship is built around: eight weeks of projects and automations instead of theory. Teams who need their people fluent in this within a working context tend to find a AI Masterclass format fits better.

The definition is one sentence. Learning P(x), not just P(y|x). Everything else in generative AI is engineering on top of that one idea, and once you hold it properly, the field stops feeling like magic and starts feeling like maths you could eventually do.

FAQs

1. Is ChatGPT a generative model?

Yes. ChatGPT is built on an autoregressive transformer, which models the probability of each next token given all previous tokens. Sampling from that distribution repeatedly is how it produces text.

2. Can a generative model be used for classification?

Yes, and this is one of its defining properties. Because a generative model learns P(x|y) and P(y), you can apply Bayes' rule to get P(y|x) and classify with it. Naive Bayes does exactly this, though discriminative models usually classify more accurately when you have plenty of labelled data.

3. What is the difference between a generative model and generative AI?

Generative model is the technical term for any model that learns a data distribution and can sample from it, including old ones like Naive Bayes and Gaussian mixtures. Generative AI is the commercial and popular term for the recent wave of large generative models that produce text, images, audio and video.

4. Are diffusion models better than GANs?

For image and video quality and training stability, diffusion models have largely replaced GANs since around 2021. GANs are still useful where inference speed matters, because a GAN generates in one forward pass while diffusion needs many denoising steps.

5. Do I need strong maths to understand generative models?

You need comfort with probability (joint versus conditional), basic linear algebra and the idea of a gradient. You do not need measure theory to build working models, though you will need more maths if you want to read the research papers properly.

6. Why do generative models hallucinate?

Because they are trained to produce likely outputs, not verified ones. A fabricated citation can be statistically plausible given the training distribution, so the model assigns it real probability and sometimes samples it.

Related reading

• What Is Generative AI? A Plain Explanation of How It Actually Works
• Latent Diffusion Models, Explained Properly
• How to Prevent AI Model Collapse: A Practical Guide for Anyone Training on Web Data
• The AI Learning Process, Explained Step by Step
• Mixture-of-Experts: The Architecture Behind Almost Every Frontier Model Now