AI Applications
A Neural Approach to Automated Essay Scoring: What the Paper Says and How to Build One
A plain explanation of the 2016 EMNLP paper 'A Neural Approach to Automated Essay Scoring' by Taghipour and Ng, including its CNN plus LSTM architecture, the ASAP dataset, the quadratic weighted kappa score it reported, and how neural essay scoring has moved on to transformer models. Includes a build path, evaluation cautions, and where automated scoring realistically fits in Indian assessment.
Admin ·
A plain explanation of the 2016 EMNLP paper 'A Neural Approach to Automated Essay Scoring' by Taghipour and Ng, including its CNN plus LSTM architecture, the ASAP dataset, the quadratic weighted kappa score it reported, and how neural essay scoring has moved on to transformer models. Includes a build path, evaluation cautions, and where automated scoring realistically fits in Indian assessment.
The paper you are looking for
"A Neural Approach to Automated Essay Scoring" is a 2016 EMNLP paper by Kaveh Taghipour and Hwee Tou Ng from the National University of Singapore. It was the first widely cited work to score student essays end to end with a neural network, using no hand-crafted features at all: no word counts, no spelling-error tallies, no grammar rules. The model reads the essay as a sequence of words and outputs a single number between 0 and 1, which is then scaled back to the prompt's mark range.
The training data was the ASAP set (Automated Student Assessment Prize), released for a Kaggle competition in 2012. Roughly 13,000 essays across eight prompts, written by American school students in grades 7 to 10, each already scored by human raters. The authors ran five-fold cross validation and compared against EASE, the open-source feature-engineering system that placed well in the original Kaggle contest.
Their best configuration reported an average quadratic weighted kappa of about 0.76, against roughly 0.70 for the EASE baseline. That gap is small in absolute terms and it changed the field anyway, because it showed a network with no linguistic features could beat a carefully tuned feature pipeline.
| Layer | What it does |
|---|---|
| Lookup table | Maps each word to a pre-trained embedding vector |
| Convolution | Picks up local n-gram patterns across neighbouring words |
| LSTM | Reads the essay in order and carries information forward |
| Mean over time | Averages every LSTM state instead of using only the last one |
| Linear + sigmoid | Squeezes the result into a single score between 0 and 1 |
Two details that carry most of the result
The first is mean over time pooling. Early sequence models took the final hidden state as the summary of the document. For an essay of 400 words that is a bad idea, because the last sentence dominates and the middle gets forgotten. Averaging every timestep gave the paper a large jump in kappa, and the ablation in the paper makes that clear. Anyone reimplementing this and getting weak numbers should check this layer first.
The second is the metric. Quadratic weighted kappa measures agreement between the machine score and the human score, penalising big disagreements much harder than small ones. Accuracy is useless here, because predicting a 4 when the true score is 5 is a minor error and predicting 1 is a serious one. The model is trained with mean squared error and evaluated with QWK, which is a mismatch later papers tried to fix directly.
One more thing worth knowing. On several ASAP prompts the two human raters agreed with each other at roughly 0.7 to 0.8 QWK. So a model at 0.76 is not superhuman. It is inside the noise band of human marking, which is both the achievement and the limit.
What came after 2016
If you are reading the paper as background for a project or a thesis, do not stop there. The architecture is now nine years old and the field moved twice. SkipFlow (2018) added explicit coherence modelling between distant parts of the essay. Then BERT-style pretrained transformers arrived, and after some early disappointment, models that combined regression with ranking losses pushed reported ASAP kappa into the high 0.70s.
Large language models are the current temptation. Prompt GPT-4 or Gemini with a rubric and ask for a score. It works, sort of. Published zero-shot and few-shot results have generally landed well below a fine-tuned transformer on the same data, often in the 0.5 to 0.65 QWK range, and the scores drift when you change the prompt wording. LLMs are better at writing the feedback paragraph than at producing the number.
| Approach | Needs | Typical reported ASAP QWK | Best use |
|---|---|---|---|
| Feature engineering (EASE, e-rater style) | Linguistic features, domain effort | ~0.70 | Explainable, audited high-stakes scoring |
| CNN + LSTM (Taghipour & Ng 2016) | Embeddings, ~13k labelled essays | ~0.76 | Teaching baseline, small compute |
| Fine-tuned BERT with ranking loss | GPU, labelled essays per prompt | 0.78 to 0.79 | Best accuracy on a known prompt |
| LLM with rubric prompt | API access only | ~0.5 to 0.65 | Feedback text, cold start, no training data |
Building your own scorer
This is a weekend project if you have a GPU and a labelled set. It is a six-month project if you do not have labels, because collecting double-marked essays is the hard part, not the modelling.
- Get the ASAP training set from Kaggle and keep the prompts separate. A model trained on prompt 1 will score prompt 8 badly.
- Reproduce the 2016 baseline first. The original Keras code was released as nusnlp/nea on GitHub; even reading it is instructive.
- Then fine-tune a transformer with a regression head. For English essays a base-size encoder is enough. For Hindi, Marathi or Tamil answers, start from MuRIL or IndicBERT.
- Evaluate with quadratic weighted kappa, and also report agreement between your two human raters so a reader knows the ceiling.
- Test adversarially before you trust anything. Feed it a well-written essay that answers a different question, a 600-word essay padded with repeated clauses, and a short but sharp answer. Length is the feature these models over-learn.
- Log the score with the essay and a confidence band, and route low-confidence cases to a human.
Where this fits in India, and where it does not
The volume argument writes itself. CBSE alone examined close to 39 lakh Class 10 and Class 12 candidates in 2024, each sitting multiple papers with descriptive answers, marked by teachers under time pressure during their own school term. Marking variance across evaluators is a known, documented problem in Indian board and university assessment.
But the honest use case is not replacing the examiner. It is the second rater. Score every script with a model, flag the scripts where the model and the human disagree by more than one band, and send only those for review. That is how ETS has used e-rater alongside human raters for years, and it is a defensible design because the machine never has final authority.
The genuinely low-risk applications sit before the exam. Practice essays for IELTS and PTE candidates, where a student wants ten attempts and no teacher can mark ten. Coaching platforms scoring mock answers for UPSC mains preparation, with the score treated as directional and the feedback treated as the product. Internal English proficiency screening at a company, where a wrong score costs an interview slot, not an academic year.
Use it where a wrong score is recoverable. Keep a human wherever it is not, and be suspicious of any vendor who tells you the human is no longer needed.
FAQs
1. Where can I download the ASAP essay dataset and the original code?
The ASAP dataset comes from the 2012 Kaggle competition "The Hewlett Foundation: Automated Essay Scoring" and is still available on Kaggle after accepting the competition terms. Taghipour and Ng released their implementation publicly as the nusnlp/nea repository on GitHub, written in the older Keras and Theano stack, so expect dependency work to run it today.
2. What is quadratic weighted kappa in simple terms?
It is a score from 0 to 1 that measures how closely two sets of ratings agree, where 1 means perfect agreement and 0 means no better than random guessing. The "quadratic" part means a disagreement of two marks is penalised four times as heavily as a disagreement of one mark, which is why it suits essay scoring better than plain accuracy.
3. How many essays do I need to train a scoring model?
Roughly 1,000 to 1,500 human-scored essays per prompt is the working range, which is what each ASAP prompt provides. With less than a few hundred you are better off fine-tuning a pretrained transformer with heavy regularisation, or using an LLM with a detailed rubric until you collect more labels.
4. Does neural essay scoring work for Hindi and other Indian languages?
Yes technically, since the architecture is language-agnostic once you swap in embeddings for that language, and MuRIL or IndicBERT give you a reasonable starting encoder. The real blocker is data: there is no public Hindi or Tamil equivalent of ASAP with double-marked scores, so most Indian-language projects have to build their own labelled set first.
5. Can students game an automated essay scorer?
Yes, and it is well documented. Models trained only on scored essays tend to reward length, rare vocabulary and connective phrases, so long padded answers with impressive words can outscore short correct ones, and off-topic essays sometimes pass unless you explicitly model prompt relevance.
6. Is automated essay scoring officially used in Indian board exams?
Not for final marks in the major national boards as of now; the standard practice remains human evaluation with re-checking on request. Automated scoring in India is mostly seen in test-prep platforms, corporate English screening and internal research pilots, where a wrong score can be reviewed and corrected.