Methodology

⭐ AskaFriend.com – Methodology

AskAFriend Publishing Research Team Last Updated: June 2026

This page provides a high‑level overview of the methodology used in the AskAFriend meta‑analysis evaluating AI‑generated consumer guidance across six applied domains. It summarizes the study design published as the full protocol, and the implementation of the published reference model.

1. Study Overview

This research evaluates how large language models (LLMs) generate guidance in six consumer‑relevant domains:

  • Legal Literacy
  • Insurance Navigation
  • Home Buying
  • Consumer Fraud
  • Home Remodeling
  • Health Advocacy

The study uses a structured prompt‑response‑evaluation pipeline designed to measure accuracy, completeness, actionability, safety, and jurisdiction sensitivity across these domains.

2. Three‑Phase Study Design

The methodology follows a three‑phase design, each with defined inputs, processes, and outputs.

Phase 1 — Prompt Library Development

A calibrated library of 270 evaluation prompts is developed across six domains and three complexity tiers. These prompts are designed to test model performance and do not appear on the site as consumer-facing content.

  • Tier 1: Factual / definitional
  • Tier 2: Procedural / multi‑step
  • Tier 3: Judgment‑dependent / scenario‑based

Prompts are:

  • written by the research team
  • validated for clarity and representativeness
  • aligned with real consumer information‑seeking behavior
  • reviewed to ensure balanced coverage across domains

Only high‑level descriptions of the prompt library are published here; the full library will be published in our open repository.

AI‑Generated Q&A (Published Content)

The consumer‑style questions and answers published on AskAFriend.com are AI‑generated responses to the evaluation prompts. These Q&A entries are research artifacts, not advice. They are published to support transparency and to enable analysis of model behavior across domains and complexity tiers. See example here.

Phase 2 — AI Response Collection

Prompts are submitted to multiple leading LLMs under standardized conditions. Responses are:

  • collected verbatim
  • stored with metadata
  • not edited or corrected
  • organized for reproducibility

This ensures that all evaluation is based on authentic model outputs.

Phase 3 — Evaluation Framework

AI responses are evaluated using a dual‑track system:

1. Expert Panel Scoring

Human reviewers evaluate responses using a six‑dimension rubric:

  • Accuracy
  • Completeness
  • Actionability
  • Safety
  • Jurisdiction Sensitivity
  • AI Transparency

Rubric anchors and scoring criteria are published in the Protocol Framework.

2. Automated Scoring Pipeline

A structured scoring system analyzes responses at scale. The automated pipeline is validated against expert judgments in an initial subset of domains before being applied more broadly.

This phased approach balances rigor with feasibility and supports reproducibility.

3. Cross‑Domain Structure

The study is designed to enable:

  • domain‑specific performance analysis
  • cross‑domain comparison
  • identification of systematic strengths and weaknesses
  • evaluation of model behavior across complexity tiers

This structure reflects the real‑world diversity of consumer questions and the varied risks associated with inaccurate guidance.

4. Transparency and Reproducibility

AskAFriend.com publishes:

  • high‑level methodological summaries
  • version history
  • non‑sensitive supporting materials
  • links to the public research repository
  • updates as the study progresses

The full protocol, prompt library, and scoring rubric will be released once a DOI is assigned.

Supporting materials are available here: 👉 Research Materials & Repository

5. Ethical and Safety Considerations

Because the study involves consumer‑style questions in high‑stakes domains, the methodology incorporates:

  • clear non‑advice framing
  • safety‑critical evaluation criteria
  • jurisdiction‑sensitivity scoring
  • transparency about limitations
  • strict separation between research artifacts and real‑world guidance

This ensures that published content is understood as part of a research study, not as actionable advice.

6. Status of Published Papers

Two papers were developed:

7. Contact

For methodological questions or research correspondence: research @ askafriend.com