In everyday statistical work, artificial intelligence often means a program that learns from past examples to sort messy text into official categories. Statistics Canada built such models in 2020 to assign industry, occupation, and class-of-worker codes to answers in its monthly national jobs survey, trained them on about 1.5 million earlier records, and put them into survey production in October 2021. The models code only the answers they are confident about, and people still check a sample of the output.
Key takeaways
- The task is narrow: reading a respondent's job description and choosing a code from one of 324 industry classes, 500 occupation classes, or two worker classes.
- The model returns a confidence score between 0 and 1, and Statistics Canada accepted predictions only above a chosen threshold and only for classes that reached 95 percent precision.
- On the validation year, about 18.5 percent of records could be coded fully by the models, with record-level precision of 98.2 to 99.0 percent.
- The authors expected the models to move coders from first-pass coding toward checking harder records, not to remove coders.
The job: turning a sentence into a code
Statistics Canada's monthly jobs survey, abbreviated LFS in the paper, is the source of Canada's employment and unemployment figures. Each month, respondents describe their work in their own words, and staff convert those descriptions into codes from three standard systems: the North American Industry Classification System (NAICS) for the employer's industry, the National Occupational Classification (NOC) for the job, and the Primary Class of Workers (PCOW) for public or private employment. According to the paper by Justin Evans and Javier Oyarzun in the Statistics Canada Symposium 2021 proceedings, coders manually code 15,000 to 20,000 records in about 7.5 days, and up to 20,000 records are coded by hand each month. The time pressure and volume led the agency to look for a machine learning solution.
How the program learns
The method is supervised learning: show a program many records that people have already coded, and let it find the patterns that link wording to codes. The team used fastText, a text classification library described in a 2016 research paper by Armand Joulin and colleagues. FastText breaks each word into short character fragments, so a misspelled or run-together word still resembles the correct one. The authors note that survey answers commonly contain misspelled words and industry-specific terms, and that fastText produced high-quality predictions on them.
Training data came from January 2013 to May 2019 (1,500,000 records). To test the models fairly, the team kept one full year, June 2019 to May 2020 (200,000 records), aside as a validation set, so the models were scored on answers they had never seen. Using a full year also covered seasonal changes in the jobs people report.
Deciding when to trust a prediction
For each record the model proposes the highest-scoring class and a confidence score. The paper tested two ways to use that score and reports the results in an options table.
| Option | Rule | Share of records coded | Overall precision | Classes above 95% precision |
|---|---|---|---|---|
| 1 | Confidence of at least 0.88 | 62.08% | 95.10% | 100 |
| 2 | Confidence of at least 0.96, and drop classes below 95% precision | 41.41% | 98.29% | 143 |
Option 1 coded more records, but many individual classes fell below an acceptable level for official statistics, so the agency chose Option 2. Higher precision cost a lower coding rate. The paper's results on the validation year show the effect of that choice: the models accepted 40.9 percent of NAICS predictions and 30.7 percent of NOC predictions at the single-classification level, with precision of 98.3 and 97.5 percent. Records fully coded by the models, meaning the model supplied the codes, made up about 18.5 percent of the validation year, with precision between 98.2 and 99.0 percent depending on the classification.
Checking that the published numbers do not shift
A coding change could alter trends without any real change in employment, so the team tested for that. Among records the models would have coded fully, the disagreement rate with human coders was about the same as the disagreement between two human coding platforms: 2.2 percent for NAICS in the comparison month, against 2.2 percent and 1.7 percent for the model versus each platform. In quality control, a random sample of model-coded records showed an error rate of 0.9 percent for NAICS and 1.2 percent for NOC.
The authors also compared 13,665 estimates, spread across NAICS, NOC, and PCOW classes, between a manual-only process and a mixed process over October 2019 to September 2020. They found no structural breaks, and they expected some volatility in classes with few records. The paper reports that machine learning coding was implemented in LFS production in October 2021.
What changes for the people involved
The paper models the coding center's capacity at roughly 24,200 records a month. In its scenarios, about 3,000 records would be coded by the model. If staff time freed that way is reinvested in more quality control on the harder remaining records, the paper estimates the outgoing error rate would fall from 14.78 percent to 10.42 percent with 50 percent checking of model output, and to 9.51 percent with 10 percent checking. These are modeled estimates from the authors, not measured outcomes. They support the paper's conclusion that machine learning shifts the nature of manual coding work rather than ending it.
What this example shows about AI in general
- The system learns from labeled examples, so it can only reproduce the categories and habits found in its training records.
- It is limited to one task with a fixed list of answers, unlike a chatbot that produces open-ended text.
- A confidence score is a design tool: an agency can trade coverage against accuracy by moving a threshold.
- The evidence is a published evaluation with numbers, including its limits, such as classes the model could not code reliably.
- Human review stays in the loop, with a sampled quality-control process applied to the machine's output.
The figures describe the 2020 to 2021 models and the validation year used in the paper. Whether Statistics Canada has since changed the models, thresholds, or coverage was not confirmed, and the article does not describe current rates.
FAQ
Is this the same kind of AI as a chatbot?
No. The models classify text into a fixed set of codes. They do not write answers, and the paper does not describe a generative system.
Why not let the model code every record?
Many classes did not reach the 95 percent precision target. Statistics Canada kept those with coders, because errors in official statistics can distort published estimates.
Did the change affect published employment figures?
In the paper's tests, 13,665 estimates showed no structural breaks between manual and mixed coding, although classes with few records can be volatile.
Is the model's confidence score a probability?
The paper describes it as only weakly associated with the model's probability of being right, and it says calibration methods could tune the scores in future work.
Related reading
- AI applications in organizations
- What AI is: definition and implications
- Why AI is considered beneficial
- How artificial neural networks work
More in AI Basics.
Last verified: October 2026
Sources
- Justin Evans and Javier Oyarzun, paper on fastText coding of the monthly national jobs survey ("Need for Speed"), Proceedings of Statistics Canada Symposium 2021, Statistics Canada, November 5, 2021
- Bag of Tricks for Efficient Text Classification, Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov, arXiv, 2016