RedLectern topic
Methods for analyzing and processing language beyond conversational LLMs.
For: Learners · developers.
After this page you can: Choose an approach to a language-processing task.
Page reviewed
Part of: Artificial Intelligence › AI foundations
Natural language processing (NLP) applies computational methods to human language. A text-processing program can manipulate and analyze language data, while an NLP system can be designed for a particular output, such as a category, a label for each word, a sentence structure, or a probability for a word sequence.
Choose an approach by starting with the output your task needs. Use classification when an input needs a category; use sequence labeling when each token needs a label; use parsing when you need sentence structure. A language model estimates probabilities for likely words or word sequences. These approaches can be combined, but they answer different questions.
For a first project, write down the input, the output, and a few representative examples. Then choose a method whose output matches the task. If the task depends on distinctions such as punctuation, capitalization, or word boundaries, check how the text is represented before selecting or training a method.
Start with the introductory examples, then try a small text-analysis exercise.
Start with: Natural Language Processing with Python: Language Processing and Python
Learn how raw text can be split into tokens and prepared for analysis.
Start with: Natural Language Processing with Python: Processing Raw Text
Start by naming the output you need. A document classifier assigns one or more category labels to an input; supervised classification learns from examples paired with labels. A language model instead assigns probabilities to likely next words or complete word sequences. These approaches answer different questions: use classification when the required output is a label from a task's label set, and consider a language model when the task requires estimating likely word sequences.
Write down whether the input is a document, sentence, or token sequence, and whether the requested output is a class, token labels, structure, or probability. That small specification helps distinguish tasks that can otherwise sound similar.
Explains language models as probability models for words and word sequences.
Natural language processing
Computational methods for analyzing, understanding, or generating human language.
Language model
A model that estimates patterns or probabilities in language.
For each request, choose a likely output type: mark each word as a noun or verb; label an email as spam or not spam; estimate which word is likely to follow “peanut butter and”.
The first needs token labels, the second needs a class label, and the third needs a probability over likely next words.
Before applying a method, decide how the text should be represented. Tokenization splits text into units such as words and punctuation; the resulting units are the input to later analysis. A document may also contain material, such as markup or headers, that is not part of the text you mean to analyze.
Preprocessing choices affect what information remains available. For example, lowercasing collapses distinctions between capitalized and lowercase forms, and removing punctuation discards punctuation tokens. Inspect a few examples from your data and keep only the transformations that fit the task. A word-counting exercise, a named-entity task, and a sentence-structure task may need different text handling.
Introduces tokens, vocabulary, and basic ways to examine text.
You want to count how often the words “Maple” and “maple” occur, treating them as the same form. What preprocessing choice would help, and what distinction would it remove?
Lowercase the tokens before counting. This merges capitalized and lowercase forms, so the count no longer preserves that distinction.
When the output belongs to individual words, use a tagging or sequence-labeling approach: a part-of-speech tagger attaches a word-class label to each word, and named-entity recognition identifies and labels entity mentions. When the output needs to show how a sentence is organized, use parsing. These outputs represent different levels of analysis, so choose according to what a later step needs to consume.
Some tasks need structured information rather than a full sentence analysis. Information extraction can identify entities and relations and represent them as structured data. For a small, defined set of questions, a system can also map sentence forms to meaning representations that can query a database. Begin with the narrowest output that answers your task.
Explores grammar, parsing, and structural ambiguity in sentences.
Shows how text can be turned into structured entities and relations.
Introduces meaning representations and a restricted language-to-database example.
A text tool must return every organization name and the location related to it. Should its immediate output be part-of-speech tags, a parse tree, or structured entity-relation records?
Structured entity-relation records fit the requested output: organization, relation, and location. Part-of-speech tags and a parse tree represent different kinds of linguistic structure.
No recent updates are available.
Jacob Eisenstein · 2019
A technical introduction used for text preparation, classification, and representation methods. The author’s draft is available online; the University of Edinburgh notes it assumes comfort with formal methods.
Daniel Jurafsky and James H. Martin · 2026
A broad third-edition draft covering text classification, tagging, parsing, and other language tasks. The authors provide the manuscript online, and the University of Edinburgh assigns it as required reading.
Christopher Manning and Hinrich Schütze · 1999
A substantial statistical treatment spanning language models, tagging, parsing, and text categorization. Its companion site lists chapters and sample materials.
Steven Bird, Ewan Klein, and Edward Loper · 2009
A hands-on introduction with programs, examples, and exercises for analyzing text with Python and NLTK. The authors provide a free online version; its age makes it a practical supplement.