To create a candidate assessment test, first define what the person needs to do well in the role, choose the question type that best reflects this, and decide how each answer will be scored before administering it. Then, test it with employees who already hold the position and use it alongside, not in place of, validated psychometric tests.
This guide explains when a custom test is worthwhile, how to design and score it, and how to combine it with validated tests without losing rigor.
Table of Contents
When is it appropriate to create your own test?
A custom test makes sense when what you want to measure is specific to your company or industry and no standard test covers it. For psychological traits, such as personality or reasoning, a validated test is advisable: measuring them accurately requires reliability, validity, and norm studies that a company can rarely conduct on its own.
| What do you want to measure? | Own proof | Validated test |
|---|---|---|
| Knowledge of your internal processes, products, or systems | Yes: no one else knows him | It doesn't exist. |
| Resolving real-life cases related to the position (a claim, an accounting close, a proposal) | Yes, as a work sample | Sometimes, as a general case |
| Minimum requirements (certification, availability, license) | Yes, as filter questions. | Not applicable |
| General technical skills (Excel, SQL, English) | Only if there is no reliable test | Yes, it usually exists |
| Personality, reasoning, soft skills | No | Yeah |
The fact that it is one's own does not exempt it from rigor. International Test Commission test user guide It requires that any test, whether a questionnaire or a work sample, have evidence of reliability and validity for its intended use. In a test of your own, that evidence begins with careful design and a pilot test.
Step 1: Define what you are going to measure
Part of the job profileIdentify the critical tasks, what distinguishes high performers, and the non-negotiable requirements. Talk to the direct supervisor and two or three top-performing employees. Use this information to create a table of specifications: what aspects you will evaluate, the weight given to each, and the type of questions you will ask.
An example for an accounts payable analyst position:
| Aspect to be evaluated | Weight | Question type | Questions |
|---|---|---|---|
| Knowledge of the company's payment process | 30 % | Multiple choice | 10 |
| Reconciling an invoice with its purchase order | 40 % | Case with attached document | 1 |
| Criteria for dealing with an exception (duplicate invoice, supplier not registered) | 30 % | Open with rubric | 2 |
Limit yourself to three to five aspects. A test that tries to measure everything becomes lengthy, tires the candidate, and doesn't thoroughly assess anything. Leave out anything unrelated to the job.
Step 2: Choose the question type
Each format serves a different purpose and has a different qualification cost:
| Question type | What shows well | How to rate |
|---|---|---|
| Multiple or single choice | Concrete knowledge, criteria among alternatives | Automatic, with answer key |
| True or false | Basic concepts | Automatic; with only two options, guessing randomly is easy. |
| Open | Reasoning, argumentation, writing | With a rubric; requires time from an evaluator |
| Case with attached document | Actual work: analyze, correct, propose | With rubric by criteria |
| Scales and matrices | Domain Preferences or Self-Assessment | By score; watch out for social desirability |
| Video | Oral communication, handling situations | With a rubric, by one or more evaluators |
Multiple choice does not require five alternatives. The meta-analysis of Michael Rodriguez on 80 years of research He concluded that three well-constructed options do not worsen the psychometric quality and allow for more questions to be asked in the same amount of time.
Step 3: Write questions that measure what matters
- One idea per question. If the question mixes two concepts, you won't know which one was wrong.
- Job context. Better than a textbook definition: «An invoice arrives without a purchase order; what do you do first?».
- Believable distractions. Incorrect choices should be mistakes that someone in the position could make, not absurd answers.
- No cheating. Avoid double negatives, words like «always» or «never» that give away the answer, and cultural references that not everyone shares.
- Clear instructions. State what is expected, how much time is allowed, and how it will be graded. Only set a time limit if speed is part of the job.
- Question bank. Write more questions than you'll use and rotate versions. This reduces the risk of answers circulating. The guide to the knowledge test It explains how to calibrate your level and time.
Step 4: Decide how it will be graded before applying it
The rating is determined before the first response is seen. If a decision is made afterward, the evaluator adjusts the criteria to suit the candidate they liked.
For closed-ended questions, an answer key reviewed by someone proficient in the role is sufficient. For open-ended questions, case studies, and video, use a rubric with described levels. Here's an example of the criteria for handling exceptions:
| Level | Description |
|---|---|
| 1. Insufficient | Pay or reject the bill without verifying anything. |
| 2. Acceptable | It detects the problem and escalates it, without proposing a solution. |
| 3. Solid | Identify the problem, check with the correct department, and propose ways to prevent it from happening again. |
Before using the rubric, ask two evaluators to separately rate the same five or six responses and compare them. If they disagree, the rubric is ambiguous and the descriptions need to be refined.
It also establishes how the decision is made: a minimum score per critical aspect works better than a single overall score, because it prevents a strength from hiding a deficiency that the position does not tolerate.
Step 5: Conduct a pilot test and review
Apply the test to several employees who already hold the position, ideally with different performance levels. Review:
- Questions that everyone gets right or no one gets right. They don't distinguish anyone; change them or remove them.
- Questions that even the best get wrong. They are usually poorly written or have the wrong key.
- Real time. If almost no one finishes and speed isn't what you're measuring, extend the time or cut questions.
- Comments. Ask what wasn't understood.
Once in use, review it periodically: compare the results of those you hired with their subsequent performance and update the questions when processes change.
How can it be combined with validated psychometric tests without losing rigor?
The most useful combination is usually a test tailored to the specific job and psychometric tests for work validated for what requires a well-supported instrument, such as reasoning or soft skills. To ensure the battery maintains its rigor:
- Report each result separately. A validated test is interpreted against a benchmark (how you measure up to a reference group); a self-administered test, against the criteria you set (what you know about the job). Adding them together into a single number mixes scales that are not comparable.
- Set the weights before the process. Decide how much each test counts based on the profile and apply it equally to everyone.
- Don't measure the same thing twice. If you already have a validated Excel test, your own test should focus on what that one doesn't cover, such as your templates or your process.
- The interview is also structured. The review of Sackett and colleagues (2022) The study identified the structured interview as the selection procedure with the highest estimated validity and concluded that the validity of many tests is lower than previously thought. The lesson applies to your test: use the same questions for everyone and grade based on predetermined criteria.
- The decision is human. The results organize the information; the decision-maker reviews the entire set, along with the CV and the interview.
In Hirint, the custom tests They allow you to introduce your own tests or design them from scratch with open-ended, multiple-choice, or single-choice questions, true or false questions, document uploads, images, scales and matrices, video, and AI-analyzed open-ended questions, with a report of the result for each answer. Assessment Architect It helps define what to measure for each role, suggests the most predictive tools, and weights skills with transparent rules, always with final human validation. Before creating a technical test, check if one already exists in the library, such as the interactive Excel test (20 minutes).
What should be considered in terms of legal aspects and the candidate's experience?
This is general guidance, not legal advice; review your case with your legal department.
- Non-discrimination. In Mexico, the Article 133 of the Federal Labor Law It prohibits refusing to accept workers based on ethnicity, gender, age, disability, health conditions, religion, marital status, or other discriminatory criteria. It ensures that no questions address these topics and that the test can be taken with adjustments if needed.
- Personal data. The Federal Law on the Protection of Personal Data Held by Private Parties, The regulation, published in March 2025, requires informing individuals about data processing through a privacy notice that identifies sensitive data and its purposes. Include only what you need to evaluate.
- Respect for the candidate's time. A lengthy test at the beginning of the process causes the most promising candidates to drop out. Adjust the duration to the stage: short for filtering, more comprehensive for finalists.
- Feedback. Explain what was evaluated and, if possible, share a summary of the result.
Common mistakes when creating your own tests
- Copying questions from the internet that are not related to the position.
- Grade the open-ended questions «at the discretion of the task» without a rubric.
- Use your own test to measure personality or intelligence.
- Giving more weight to the test than to the rest of the process.
- Never check it after the first use.
Frequently Asked Questions
How many questions should a candidate assessment test have?
It depends on how many aspects you're measuring and the type of question. As a practical guideline, several closed-ended questions for each aspect and one or two open-ended questions or case studies for those requiring reasoning. More important than the number of questions is that each aspect has enough questions so that it doesn't depend on a single one, and that the test can be completed at a leisurely pace if speed isn't what you're measuring.
How to avoid bias when creating your own test?
Write questions that are relevant to the job responsibilities, avoid cultural references and personal topics, and define the rubric before administering the test. Calibrate the evaluators with sample responses and check if any group of candidates consistently scores differently without justification for the position. If discrepancies arise, review the questions involved.
Can I use artificial intelligence to create the questions?
Yes, it can help generate drafts or variations, but each question should be reviewed by someone with expertise in the role: AI can suggest correct or incorrect answers, or questions unrelated to your context. The answer key, rubric, and final decision remain the responsibility of your team.
Does a proprietary test replace psychometric tests?
No. A proprietary test can accurately measure knowledge and tasks specific to your company, but it lacks the norms, reliability studies, and validity that support a psychometric test. The most reliable approach is to combine them: your own test for job-specific skills and a validated test for reasoning, personality, or soft skills, each with its own weight and interpretation.
How often do you need to update your own test?
When processes, systems, or job profiles change, and also periodically even if nothing has changed, take the opportunity to compare the results of hired individuals with their performance, remove questions that are no longer relevant, and replace those that may have been missed.
If you want to see how to create your own tests in Hirint and what validated tests its library has, we'll show you using a job title you're currently filling.