Progress

0 of 8 answered

1 · Task Definition

Can you write down: 'this AI system will be considered successful if it gets the right answer X% of the time, with the failure mode being Y, and the cost of failure being Z'?

We can't fill in any of those blanks yetWe can describe the goal but not specific acceptance criteriaWe have rough numbers but they haven't been validatedYes — all three are defined and stakeholders have signed off

2 · Task Definition

Do you have an evaluation test set — real examples with correct answers labeled — that you can use to measure accuracy?

No — we'd evaluate by intuition or vibesWe could pull together examples but they aren't labeled or curatedWe have a labeled test set of 20-50 examplesYes — a curated set of 50-200+ representative examples with verified labels

3 · Data Readiness

If the AI needs to ground its answers in your existing data (RAG, fine-tuning, examples), is that data accessible and structured?

Data exists but is scattered, unstructured, or hard to extractMost of the data is accessible; some cleanup neededData is accessible and mostly clean; some structure work remainsData is accessible, structured, and already in a format the AI can consume

4 · Data Readiness

How confident are you in the quality and currency of the underlying data?

Quality is unknown or known to be poor; outdated in placesQuality varies by source; some known issues we haven't addressedQuality is generally reliable with known caveatsData quality is monitored; refreshed on a schedule; issues caught fast

5 · Failure Cost

What happens when this AI produces a wrong answer?

Financial loss, legal exposure, or harm to a customerOperational disruption that takes hours/days to resolveAnnoyance — easy to catch and correct, no lasting impactGenuinely no consequence — outputs are reviewed before any action

6 · Failure Cost

If this AI fails silently (produces wrong answers that look right), how quickly would you find out?

Maybe never — we'd assume the outputs are right and act on themEventually, when something downstream breaksWithin days through spot-checks or aggregate metricsImmediately — outputs flow through a human review queue or automated validation

7 · Operational Readiness

Is there a planned human review surface for low-confidence or high-stakes outputs?

No — outputs go directly to consumers or downstream systemsVague plan to review periodically, but no specific surface designedDesigned but not yet built — queue + review UI on the roadmapYes — review queue, escalation paths, and feedback loop into the test set

8 · Operational Readiness

Do you have visibility into expected cost-per-task at production volume, with alerts for cost overruns?

No idea what this will cost at production volumeRough estimate but no monitoring or alertsEstimate is modeled; cost monitoring planned but not builtCost-per-task is measured continuously with alerts; budget caps in place

Answer all 8 questions to see your risk profile

What it measures

What This Assessment Measures

4 dimensions, 2 questions each. Each predicts a different way AI projects fail.

Task Definition

Whether the success criteria are clear and measurable. Most failed AI projects can't fill in the 'success looks like X% accuracy with failure mode Y costing Z' template.

Data Readiness

Whether the data the AI needs is accessible, structured, and trustworthy. Often the data work IS the project; AI is a thin layer on top.

Failure Cost

How costly a wrong AI answer is, and whether the project tolerates that. Loud failures are easier to manage than silent ones.

Operational Readiness

Whether human review, monitoring, and cost controls are planned from day one. The pieces that turn a working prototype into a system that stays running.

When to use

When to Use This

→Before kicking off a new AI initiative — to catch foundational gaps before they become expensive
→Mid-project, when something feels off but the team can't articulate why
→When evaluating a vendor pitch claiming AI will solve a specific business problem
→As a structured conversation with stakeholders who are pushing for AI without specifics
→To justify slowing down a project that has executive enthusiasm but missing pieces

What it measures

What This Assessment Measures

4 dimensions, 2 questions each. Each predicts a different way AI projects fail.

Task Definition

Whether the success criteria are clear and measurable. Most failed AI projects can't fill in the 'success looks like X% accuracy with failure mode Y costing Z' template.

Data Readiness

Whether the data the AI needs is accessible, structured, and trustworthy. Often the data work IS the project; AI is a thin layer on top.

Failure Cost

How costly a wrong AI answer is, and whether the project tolerates that. Loud failures are easier to manage than silent ones.

Operational Readiness

Whether human review, monitoring, and cost controls are planned from day one. The pieces that turn a working prototype into a system that stays running.

When to use

When to Use This

→Before kicking off a new AI initiative — to catch foundational gaps before they become expensive
→Mid-project, when something feels off but the team can't articulate why
→When evaluating a vendor pitch claiming AI will solve a specific business problem
→As a structured conversation with stakeholders who are pushing for AI without specifics
→To justify slowing down a project that has executive enthusiasm but missing pieces

Is Your AI Project Set Up to Succeed?

Can you write down: 'this AI system will be considered successful if it gets the right answer X% of the time, with the failure mode being Y, and the cost of failure being Z'?

Do you have an evaluation test set — real examples with correct answers labeled — that you can use to measure accuracy?

If the AI needs to ground its answers in your existing data (RAG, fine-tuning, examples), is that data accessible and structured?

How confident are you in the quality and currency of the underlying data?

What happens when this AI produces a wrong answer?

If this AI fails silently (produces wrong answers that look right), how quickly would you find out?

Is there a planned human review surface for low-confidence or high-stakes outputs?

Do you have visibility into expected cost-per-task at production volume, with alerts for cost overruns?

What This Assessment Measures

Task Definition

Data Readiness

Failure Cost

Operational Readiness

When to Use This

Practical AI Implementation Guide

AI Integration & Automation

Is Your AI Project Set Up to Succeed?

Can you write down: 'this AI system will be considered successful if it gets the right answer X% of the time, with the failure mode being Y, and the cost of failure being Z'?

Do you have an evaluation test set — real examples with correct answers labeled — that you can use to measure accuracy?

If the AI needs to ground its answers in your existing data (RAG, fine-tuning, examples), is that data accessible and structured?

How confident are you in the quality and currency of the underlying data?

What happens when this AI produces a wrong answer?

If this AI fails silently (produces wrong answers that look right), how quickly would you find out?

Is there a planned human review surface for low-confidence or high-stakes outputs?

Do you have visibility into expected cost-per-task at production volume, with alerts for cost overruns?

What This Assessment Measures

Task Definition

Data Readiness

Failure Cost

Operational Readiness

When to Use This

Practical AI Implementation Guide

AI Integration & Automation