EXP-01 — M1 Live Weather (Validation Prototype)

Knowledge ▸

Challenge

FROM: Product
Subject: Users want live weather in chat

Beta users ask "What's the weather in Seattle?" and "Do I need an umbrella today?" Copilot gives generic, outdated answers. We need live weather this week.

Your role: AI engineer on Copilot v0.1 · Ships: This week

Constraints — your architecture must satisfy all of these

Product will not accept a solution that breaks any row below. Use these when you compare approaches.

1. Speed — under 3 seconds end-to-end From the user sending “What’s the weather in Seattle?” to Copilot’s reply must take less than 3 seconds. Slow pipelines fail this release.
2. Any city — not a fixed list Users can name Seattle, Detroit, Tokyo, or any city. The solution must work for cities you did not hard-code.
3. Live conditions — not training data Answers must reflect right now (temperature, rain, etc.). Generic facts from model memory (“Seattle is often rainy”) do not count — you must prove the data is current.
4. Small scope — ship weather in chat this week Copilot v0.1 is chat-only today. Your deliverable is live weather inside that chat — not a separate project (agent platform, memory system, doc pipeline). You should compare approaches like prompt, tool/API, and RAG; pick what best meets constraints 1–3 for this release.

Done when: "Weather in Seattle?" → current answer + evidence

Knowledge ▸

Compare

Name two ways this could work before you test. Which fails first under the freshness constraint?

Pinned constraints

Speed < 3s Question → answer in under 3 seconds
Any city Not hard-coded to Seattle only
Live conditions Prove right-now weather — not model memory
Small scope Ship weather in chat this week — compare all options, don’t expand product surface

A: Prompt engineering only

Select all that apply for this approach.

Pros Fast — no API or new infra Stays inside the LLM Easy to iterate on wording

Cons Cannot prove weather is current May hallucinate for unfamiliar cities No source or timestamp to show users

Risks under constraints Fails live-conditions constraint Sounds right but may be wrong

B: Tool / API call

Select all that apply for this approach.

Pros Fetches live data with proof Tool trace shows city and timestamp Works for any city the API supports

Cons Requires API integration Need tool schema and error handling Depends on external service uptime

Risks under constraints API latency could break 3s budget Wrong city passed to tool

C: RAG over docs (optional)

Optional third approach — select if you considered it.

Pros Can cite retrieved sources Useful when answers live in documents

Cons Weather docs may be stale Retrieval adds latency Not a live weather API

Risks under constraints Fails live-conditions constraint Likely exceeds 3s latency

Initial pick:

A — Prompt only B — Tool / API C — RAG

Knowledge ▸

Experiment

Run your initial pick first, then try at least one other approach before deciding.

Test approach

Test input: "What's the weather in Seattle?"

Decide

Pick your final architecture, then say what the experiments proved.

1. Your final decision

Prompt only Tool / API RAG

2. Your conclusion — pick at least one Evidence changed my initial pick Evidence confirmed my initial pick One approach clearly failed constraints One approach clearly met constraints

Or fill a “you tested this” rejection section below instead of this block.

Anything else? (optional)

3. Rejections — complete any sections that appear after step 1.

Prove — Transfer

You already picked an architecture. Now test it on a new city — not Seattle.

Step 1 — New city (Detroit)

Click send, read the reply and evidence panel, then check what you see.

Step 2 — Follow-up (umbrella)

Complete Step 1 first. Then send the follow-up and check whether the answer uses current Detroit weather.

Record

Participant ID

Auto-generated for this session. Facilitator can note this ID in interview records.

Decision log (auto-draft):

One line for the next engineer (optional)

M1 decision recorded

You compared approaches, ran an experiment, and proved transfer.