Prompt工程调优实录：从42%到87%准确率的三轮迭代实验

提取以下文本中的药物、疾病、疗效三元组，输出JSON。
文本：{text}

你是一个专业的医学信息抽取引擎。从输入文本中提取三元组(drug, disease, effect)。
规则：
- drug必须是具体药物名称，不含剂量
- disease必须是疾病全称，不含修饰语
- effect必须包含具体指标和数值，格式为"指标+变化方向+数值"
示例：
输入：二甲双胍在2型糖尿病患者中使HbA1c下降1.2%。
输出：{"drug": "二甲双胍", "disease": "2型糖尿病", "effect": "HbA1c下降1.2%"}
---
文本：{text}

请严格按以下JSON Schema输出，不要输出任何其他内容：
{"drug": string, "disease": string, "effect": string}
抽取完成后，请检查：effect中是否包含数字和单位？如果没有，请置为"未明确"。
文本：{text}

# prompt_engine_experiment.py
import openai
import tiktoken
import json
import time

client = openai.OpenAI(api_key="sk-xxx")
enc = tiktoken.encoding_for_model("gpt-4-turbo-0125-preview")

def run_extraction(text, prompt_template, use_json_mode=False):
    prompt = prompt_template.format(text=text)
    params = {
        "model": "gpt-4-turbo-0125-preview",
        "temperature": 0.3,
        "max_tokens": 512,
        "messages": [{"role": "system", "content": "你是精准的医学信息抽取器。"},
                     {"role": "user", "content": prompt}]
    }
    if use_json_mode:
        params["response_format"] = {"type": "json_object"}

    start = time.time()
    resp = client.chat.completions.create(**params)
    latency = time.time() - start
    content = resp.choices[0].message.content
    prompt_tokens = resp.usage.prompt_tokens
    completion_tokens = resp.usage.completion_tokens

    # 尝试解析JSON，失败则标记
    try:
        result = json.loads(content)
    except:
        result = {"error": content[:200]}

    return result, prompt_tokens, completion_tokens, latency

# 计算token消耗
def count_tokens(prompt):
    return len(enc.encode(prompt))

# run_experiment.py
test_samples = load_annotated_data("test_200.jsonl")  # 每条含text和gold_dict
prompts = {
    "zero_shot": zero_shot_template,
    "few_shot": few_shot_template,
    "schema_selfcheck": schema_selfcheck_template
}

results = {}
for name, template in prompts.items():
    f1_list, token_list = [], []
    for sample in test_samples[:20]:  # 先跑20条快速验证
        for rep in range(5):
            pred, pt, ct, lt = run_extraction(sample["text"], template, 
                                               use_json_mode=(name=="schema_selfcheck"))
            f1 = compute_f1(pred, sample["gold"])
            f1_list.append(f1)
            token_list.append(pt + ct)

    avg_f1 = sum(f1_list) / len(f1_list)
    avg_tokens = sum(token_list) / len(token_list)
    variance = statistics.variance(f1_list)
    results[name] = {"avg_f1": avg_f1, "avg_tokens": avg_tokens, "variance": variance}
    print(f"{name}: F1={avg_f1:.3f}, tokens={avg_tokens:.0f}, var={variance:.4f}")