Trust and quality notes
- Last updated
- August 26, 2026
A metric moves unexpectedly and the questions begin immediately. Is the change real? Did tracking break? Which customer segment caused it? Should the team respond now or wait for more data? Without a structured investigation, analysts can spend hours producing charts while stakeholders form conclusions from the first plausible explanation.
An anomaly investigation plan should establish what changed, test whether the change is trustworthy, and define an efficient sequence of analysis. It should not pretend to know the cause before the work begins.
Why ordinary prompting fails
“Explain why conversion dropped” invites speculation. An AI may suggest common causes that sound reasonable but are not supported by the supplied data. It may also skip basic validation, such as checking whether the metric definition, event pipeline, reporting window, or denominator changed.
A better prompt asks for hypotheses rather than conclusions. It requires each hypothesis to be connected to a test, the data needed, and a result that would support or weaken it. It also puts data-quality checks before business interpretation.
The prompt
ROLE You are a senior analytics lead designing an investigation plan for a metric anomaly. You are not determining the root cause unless the supplied evidence proves it. REQUIRED INPUTS - Metric name and exact definition - Observed value, expected value, and comparison period - Time range and reporting timezone - Data source and refresh schedule - Relevant dimensions, such as channel, product, region, device, or customer segment - Known releases, campaigns, incidents, pricing changes, and calendar events - Recent tracking or data-pipeline changes - Available tables, dashboards, and query access STEPS 1. Restate the anomaly precisely, including magnitude, direction, period, and comparison basis. 2. Check whether the anomaly could result from definition, instrumentation, ingestion, freshness, duplication, missing data, denominator, timezone, or dashboard changes. 3. Assess whether the comparison is appropriate. Consider seasonality, day-of-week mix, holidays, sample size, and recent baseline volatility only when relevant evidence is available. 4. Break the aggregate metric into the most informative dimensions. 5. Generate a prioritized set of causal hypotheses grounded in known events or observed segment patterns. 6. For each hypothesis, define the test, required data, expected signal, and interpretation of positive, negative, or inconclusive results. 7. Order the investigation to eliminate high-likelihood and high-impact explanations efficiently. 8. Define escalation conditions and immediate safeguards, if any. OUTPUT FORMAT # Metric Anomaly Investigation Plan ## Anomaly statement - Metric and definition: - Observed change: - Comparison: - First detected: - Current confidence that the anomaly is real: ## Validation checks A table with check, reason, data needed, owner, and pass/fail criterion. ## Segmentation plan A prioritized list of cuts, explaining what each could reveal. ## Hypotheses and tests A table with priority, hypothesis, supporting evidence, test, required data, expected signal, and possible interpretation. ## Investigation sequence Numbered actions with dependencies and suggested owners. ## Escalation conditions Specific findings that should trigger product, engineering, finance, marketing, or leadership involvement. ## Open questions and missing data EVIDENCE AND UNCERTAINTY RULES - Do not present a hypothesis as a cause. - Use only supplied facts when describing releases, incidents, campaigns, or customer behavior. - Cite the relevant dashboard, query, note, or event for every evidence-based hypothesis. - Label general possibilities as “generic hypothesis, not yet supported.” - State when sample size, variance, or data quality prevents interpretation. - Do not calculate significance unless the necessary data and method are supplied.
What to provide
Start with the metric definition, not just its dashboard label. Include the numerator, denominator, exclusions, attribution rules, timezone, and refresh cadence. Provide the current value, the comparison value, and enough historical data to show normal variation.
Add a dated change log for product releases, campaigns, pricing, instrumentation, and data pipelines. If raw data is unavailable, describe which tables and dimensions an analyst can access. Screenshots can provide context, but machine-readable exports are usually more useful for validating calculations.
How to review the output
Check that data validation comes before causal analysis. If the plan begins by blaming a campaign, product release, or customer segment, it is moving too quickly.
Review whether the proposed dimensions can actually be queried and whether they match the metric’s grain. Confirm that each hypothesis has a test capable of weakening it, not merely finding supporting examples. The sequence should also be practical: quick integrity checks first, targeted segment analysis next, and expensive or cross-functional work only when earlier findings justify it.
Where it fails
The prompt cannot discover a root cause from a summary alone. It creates a plan for investigation. Execution still requires valid data, analytical judgment, and often collaboration with engineering or business teams.
Sparse data can make normal variation look meaningful. Correlated events can also remain difficult to separate, especially when several launches or campaigns occurred together. For decisions involving formal statistical inference, experimentation, or material financial reporting, use an appropriately qualified analyst and documented methods.
Practical takeaway
The fastest anomaly investigation is not the one with the most charts. It is the one that rules out bad data, frames testable hypotheses, and proceeds in a deliberate order. Try this prompt in Agentic Workers when a key metric moves, then give the resulting plan to the analyst who can validate and execute it.
<!-- X derivative: When a metric moves, do not ask AI for a cause; ask for a validation-first investigation plan with testable hypotheses and clear escalation conditions. -->