What is Demerit AI and how does disqualification work?
inca includes a built-in quality-control system that evaluates participant responses during surveys and conversational interviews.
The system provides user-configurable Demerit AI to assess open-ended responses and may assign demerit points when a response is considered low quality. Demerit points may also be assigned based on participants’ responses to certain close-ended questions. If a participant accumulates enough points, they may be disqualified.
Key terms
Before getting into the settings, here are the main terms used in inca:
● Demerit AI - The AI-based quality check that evaluates open-ended responses.
● Demerit score - The raw AI score returned by Demerit AI. This ranges from 0.0 to 1.0.
● Flagged - A response is flagged when the raw demerit score is 0.8 or higher.
● Demerit points - The points assigned when Demerit AI flags a response and the question’s demerit influence is enabled; or when participant’s answers to close-ended question(s) are flagged.
● Disqualification Control - The project-level setting that determines how many total demerit points a respondent can accumulate before being disqualified.
● Respondent Quality - The overall quality status shown in the dashboard and export:
- Good Quality
- Flagged
- Disqualified
Note: Other survey rules may terminate a participant or quota them out, but disqualification is a separate status that is based only on demerit points.
How Demerit AI works
Demerit AI evaluates the response in the context of the current dialogue only.
A dialogue includes:
● the original open-ended question, and
● any follow-up probe questions if probing is enabled
Demerit AI considers the wording of the question and the participant’s response, and it can flag responses such as:
● gibberish or keyboard smashing
● uninformative answers
● off-topic or irrelevant responses
● shallow or low-effort responses
● responses in the wrong language
● toxic or inappropriate remarks
● repeated or inconsistent responses within the same dialogue
What Demerit AI does not do
● It does not look at answers to other questions in the survey
● It does not affect probing behavior
● It does not terminate the dialogue early
● It does not replace quota logic or other survey termination logic
● It does not interact with SmartCoding
Demerit AI Instructions
Researchers can optionally provide Demerit AI Instructions to give the model extra context about how to evaluate response quality.
These instructions are used by Demerit AI when determining whether a response should be flagged.
They do not affect probing.
How open-ended demerit points work
Open-ended demerit is controlled by two settings:
- Demerit AI Turns the check on for the question or project default
- Demerit Influence Determines how many demerit points are added if the response is flagged
Demerit Influence options
● Disabled / No Influence → 0 points
● Weak → 2 points
● Normal → 5 points
● Strong → 10 points
If a question has Demerit AI enabled but the influence is set to Disabled / No Influence, the response is still scored and may be flagged, but it does not contribute points toward disqualification.
Flagging threshold
A response is considered flagged when the raw demerit score is 0.8 or higher.
How disqualification works
In the Overview page, Disqualification is controlled at the project level through Disqualification Control.
Disqualification Control options
● Strict — disqualify at 5 or more demerit points
● Normal — disqualify at 10 or more demerit points
● Disable — do not disqualify participants based on demerit points
How it works
- Demerit AI evaluates the response
- If the response is flagged, the question’s demerit influence may add points
- Those points accumulate across the survey
- When the respondent reaches the project’s disqualification threshold, they become Disqualified
If the respondent has points but has not reached the threshold, they are shown as Flagged.
If the respondent has no demerit points, they are shown as Good Quality.
Configuring Demerit AI in the inca Survey Builder
On the inca platform, Demerit AI can be configured at both the project and question level.
Project-level default
Researchers can enable Demerit AI as a project-level default.
That default is only applied if the researcher explicitly enables it.
Question-level override
For any open-ended question, the researcher can:
● inherit the project-level setting, or
● override it for that question
This means a question can have:
● Demerit AI on or off independently of other questions
● its own Demerit AI Instructions
● its own Demerit Influence
Important behavior
● Demerit AI can be enabled even if SmartProbe is not enabled
● Demerit AI does NOT affect probing, only quality scoring only
Close-ended demerit points
inca also applies demerit points automatically to certain close-ended question types. This behavior is not configurable.
At present, this applies to:
● Popup questions
● Tradeoff questions
Popup questions
A respondent may receive demerit points for a Popup question if all of the following are true:
● the question has more than 2 answer options
● the Popup contains more than 7 cards
● the respondent gives the same answer on every card
When this happens, the respondent receives demerit points equal to the number of cards in the Popup question.
This check is performed at the end of the Popup sequence.
Tradeoff questions
A respondent may receive demerit points for a Tradeoff question if all of the following are true:
● the question has more than 2 buttons
● the question has more than 7 concepts/options
● the respondent selects “Not sure” for every answer in the first stage
When this happens, the respondent receives demerit points equal to the number of concepts/options in the Tradeoff question.
How close-ended demerit affects disqualification
Any close-ended demerit points are added to the respondent’s total demerit score across the survey.
That means they can contribute to the respondent being:
● Flagged
● Disqualified
depending on the project’s Disqualification Control setting.
Reporting
These points are included in the respondent’s overall demerit total, but there is no separate user-facing control for them and no special export column dedicated to this behavior.
SmartProbe API behavior
For SmartProbe API integrations, demerit is computed automatically.
What clients receive
A variable in their 3rd-party platform (depending on integration details) which scores the dialogue-level demerit score in a range from 0.0-1.0 (with 1.0 indicating the worst responses).
inca computes the demerit score automatically, but the client decides how to use it.
For example:
● the client may use it in disqualification logic
● the client may export it only
● the client may ignore it if they do not want to act on it
The demerit score is returned at the final round of the conversation.
What appears in dashboard and export
Dashboard
The dashboard shows Respondent Quality:
● Good Quality
● Flagged
● Disqualified
The dashboard does not show the raw demerit score as a 0.0–1.0 float.
Export
Export includes demerit-related data, but the display depends on the surface.
For open-ended questions on the inca platform
Export shows a binary flag for each open-ended question:
● 1 if demerit was flagged
● 0 otherwise
It does not show the raw score as a float.
For API-based conversational workflows
The API can return the raw dialogue-level demerit score, and clients can capture it if needed.
Overall respondent status
Export and dashboard both use the total demerit points to determine:
● Good Quality
● Flagged
● Disqualified
Related Articles
What is Disqualification Control and how to set it up?
Note: This documentation is based on legacy systems and we recommend users to look at the Demerit AI and Disqualification system documentation here. The Disqualification Control allows you to control how strict or lenient you can be on the way ...How to set up an Open Ended Question and AI Probing?
Note: This documentation is based on legacy systems and we recommend users to look at the SmartProbe Studio documentation here Main use: Using an open-ended question, participants can freely type answers in their own words (as opposed to a ...How to set up an Open Ended Question and optimize probing (inca SmartProbe Studio)
Main use: Using an open-ended question, participants can freely type answers in their own words (as opposed to a pre-determined list in Choice questions), which can be in lengthy essays or short texts. A. For inca platform users, you access inca ...What is Project-Level Conversational Settings and how to set it up?
In the Question Builder, users can control and apply the same Quality Control and Smartprobe settings for all Open End questions in their study. This can be done in the Overview page using the Conversational Settings tab. Once users enable the ...How to enable Multi-Turn probing in Open Ended Questions?
Note: This documentation is based on legacy systems and we recommend users to look at the SmartProbe Studio documentation here Multi-Turn Probing In the Question Builder, click Conversation Settings and toggle on/enable Smartprobe. Then, toggle ...