Who is affected when an AI system gets it wrong?

An event assistant answers English questions well but repeatedly misunderstands short messages from visitors using a second language. Its overall score looks strong because most test messages are polished English. Whose experience has that score hidden?

Before you begin: Understand that a useful-looking model output can still be unsupported or incorrect.

Look beyond the average

Bias can enter through data collection, labels, model training, interface design, and the way results are used. In this example, the test set underrepresents people whose messages differ from the majority. A high overall score does not establish equally useful service for every group or writing style.

Add representative cases with varied phrasing, spelling, language, and accessibility needs where those reflect the intended audience. Do not infer sensitive identities from writing style just to create groups. Use appropriate, consented information and task-relevant categories.

Inspect both error rates and consequences. Misclassifying praise is different from missing a request for accessibility information. People affected by the system can reveal important failures that a developer's favorite benchmark never tests.

Collect only what the task needs

If the assistant answers public event questions, it may not need names, addresses, or private registration records. Removing unnecessary data reduces exposure and simplifies the system. Data minimization means deciding what information is actually needed before collecting or forwarding it.

Anonymizing text is not as simple as deleting a name. Dates, rare roles, locations, or combinations of details can still identify someone. Do not call a dataset anonymous merely because a direct identifier is absent.

Check where data is processed, retained, logged, and shared. The model provider, your application, and connected tools may each keep different records. A prompt saying “keep this private” does not change those storage arrangements.

Make uncertainty usable

If the event notice does not answer whether a wheelchair-accessible entrance is available, the assistant should identify the missing information and provide a legitimate way to obtain it. An invented reassurance can prevent a person from making an informed plan.

Useful uncertainty is specific: what is missing, why it matters, and what can resolve it. Repeating a generic disclaimer on every response can bury the actual issue. Design the task so people can distinguish confirmed information from a draft or inference.

Give review a real decision

“A human is in the loop” is meaningful only if the person can inspect evidence, has time and authority to reject the result, and sees what action will follow approval. A button that says “approve” beside a long untraceable answer is a weak control.

For an event announcement, show the draft alongside the source notice and highlight unresolved facts. Approval should apply to that specific version. If the text changes afterward, the previous approval should not silently authorize the new content.

Avoid misleading labels about generated content

AI-text detectors can make mistakes. A detector score is not reliable proof that a person used AI or that a statement is false. Evaluate the actual evidence, provenance, and applicable disclosure requirements instead of treating style as a definitive signal.

For your own work, be clear when an image or voice is generated if the audience might otherwise mistake it for a real record. Keep citations connected to claims, and do not invent sources or testimonials to make an output look credible.

Audit one response

The assistant says, “The venue is accessible for everyone,” but its only source gives an address and room number. Identify the unsupported claim, who may be affected, and a better response.

Next, you will build a small workflow that keeps these boundaries visible from input through review.

Further reading

The NIST AI Risk Management Framework organizes work around understanding context, measuring risks, and managing them. Datasheets for Datasets motivates documenting how data was created and where it may be unsuitable.

Continue to the next lesson.