Two short messages contain the word “great.” One praises the workshop; the other complains sarcastically. A word-count rule treats them alike. Could a model learn combinations of features instead of treating each word as an isolated vote?
Before you begin: Understand features, parameters, and loss from the machine-learning lesson.
Follow one small computation
A neural network is a collection of learned numerical transformations. A basic unit takes numbers, multiplies them by weights, adds a bias, and applies a function to the result. A weight controls how strongly an input contributes. A bias shifts the result even when the inputs are zero.
Consider two made-up features: x1 measures praise-related language and x2 measures complaint-related language. A unit might calculate z = 2 × x1 - 3 × x2 + 1. With x1 = 1 and x2 = 1, z is zero. With praise alone, z is three. These numbers are for tracing the arithmetic; real text features and weights are learned rather than assigned these neat meanings.
The unit then applies an activation function. One common function, ReLU, returns zero for a negative input and leaves a positive input alone. It introduces a bend into the computation. That bend is important.
Why stacking straight lines is not enough
If every layer only performed a linear transformation, several layers could be combined into one linear transformation. Extra layers alone would not create the range of relationships people expect from a neural network. Nonlinear activations let the model represent more complicated combinations.
Think of a simple decision: a registration qualifies for a discount when a participant has either a student code or a volunteer code, but not both. A single straight boundary over the two yes/no inputs cannot separate exactly those two qualifying cases from the others. A network with suitable nonlinear hidden units can represent that pattern.
“Hidden” simply means an intermediate layer, between input and output. It does not mean the layer contains a secret human-readable concept. Some internal patterns can be interpreted, but assigning a clear role to every unit is generally unjustified.
What makes learning deep?
Deep learning uses networks with multiple learned layers. Earlier transformations produce representations that later transformations can use. In image tasks, some models learn useful local visual patterns; in language tasks, the representation of a word can depend on the surrounding text. The exact organization depends on the architecture and training objective.
Depth is not a guarantee of quality. A network needs representative data, a suitable objective, a workable optimization method, and evaluation. For a small table of clear numeric features, a simpler model may be a better choice. Deep learning is particularly useful when learning representations from complex inputs provides an advantage.
How do the weights change?
A training example passes forward through the network to produce a prediction. The loss measures its error. Backpropagation applies the chain rule from calculus to calculate how each parameter contributes to changing that loss. An optimizer uses those gradients to update the parameters.
You do not need calculus yet to follow the idea: estimate which adjustable values should move, and in which direction, to make the training score smaller. Repeat over many batches. This is different from manually telling each hidden unit what feature it must detect.
Smaller training loss alone is not enough. The model may memorize rare examples, copy unwanted biases in the data, or find a shortcut that fails outside the training setting. Keep a separate evaluation set and examine individual errors.
Run a tiny forward pass
This complete program uses Python 3 and no extra packages. It performs a calculation with fixed weights; it does not train a model.
# Runnable: Python 3, standard library.
def relu(value):
return max(0.0, value)
def unit(praise, complaint):
return relu(2 * praise - 3 * complaint + 1)
assert unit(1, 0) == 3
assert unit(1, 1) == 0
print(unit(0, 1))
Before running it, predict the last output. Then change the bias from one to four. Which cases change, and why?
Trace the changed network
The original last output is zero: the weighted sum is negative and ReLU clips it. With a bias of four, complaint alone produces one. The unit becomes easier to activate for every input. This is a parameter change, not evidence that the new setting is more accurate. You would need labeled examples to judge accuracy.
A neural network still needs numeric inputs. The next lesson asks how words become those numbers without pretending that a word's ID already contains its meaning.
Further reading
Google's neural networks module develops activations and hidden layers. The fixed two-feature example above is illustrative, not a trained sentiment system.