e5-large
简介
**新闻（2023年5月）：请切换至e5-large-v2，该版本性能更优且使用方法相同。**
模型卡片
模型配置
模型详情
已翻译E5-large
新闻（2023年5月）：请切换至 e5-large-v2，该版本性能更优且使用方法相同。
通过弱监督对比预训练实现的文本嵌入（Text Embeddings by Weakly-Supervised Contrastive Pre-training）
梁王，杨楠，黄小龙，焦斌星，杨林军，蒋大兴，Rangan Majumder，魏富，arXiv 2022
该模型包含 24 层，embedding 大小为 1024。
使用方法
以下是对 MS-MARCO 段落排序数据集中的查询和段落进行编码的示例。
import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel
def average_pool(last_hidden_states: Tensor,
attention_mask: Tensor) -> Tensor:
last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
# Each input text should start with "query: " or "passage: ".
# For tasks other than retrieval, you can simply use the "query: " prefix.
input_texts = ['query: how much protein should a female eat',
'query: summit define',
"passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
"passage: Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments."]
tokenizer = AutoTokenizer.from_pretrained('intfloat/e5-large')
model = AutoModel.from_pretrained('intfloat/e5-large')
# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
训练细节
请参考我们的论文：https://arxiv.org/pdf/2212.03533.pdf
基准评估
请查看 unilm/e5 以复现 BEIR 和 MTEB benchmark 上的评估结果。
对 Sentence Transformers 的支持
以下是与 sentence_transformers 配合使用的示例。
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('intfloat/e5-large')
input_texts = [
'query: how much protein should a female eat',
'query: summit define',
"passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
"passage: Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments."
]
embeddings = model.encode(input_texts, normalize_embeddings=True)
包依赖
pip install sentence_transformers~=2.2.2
贡献者：michaelfeil
常见问题
1. 我是否需要在输入文本中添加 "query: " 和 "passage: " 前缀？
是的，这是模型的训练方式，否则你会看到性能下降。
以下是一些经验法则：
- 对于非对称任务（如开放 QA 中的段落检索、即席信息检索），请分别使用 "query: " 和 "passage: " 前缀。
- 对于对称任务（如语义相似度、释义检索），请使用 "query: " 前缀。
- 如果你希望将 embeddings 用作特征（如线性探测分类、聚类），请使用 "query: " 前缀。
2. 为什么我复现的结果与模型卡片中报告的结果略有不同？
不同版本的 transformers 和 pytorch 可能会导致可忽略但非零的性能差异。
3. 为什么余弦相似度分数分布在 0.7 到 1.0 之间？
这是已知且预期的行为，因为我们在 InfoNCE 对比损失中使用了较低的温度参数 0.01。
对于文本检索或语义相似度等文本嵌入任务，重要的是分数的相对顺序而非绝对值，因此这不应成为问题。
引用
如果你觉得我们的论文或模型有帮助，请考虑如下引用：
@article{wang2022text,
title={Text Embeddings by Weakly-Supervised Contrastive Pre-training},
author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Jiao, Binxing and Yang, Linjun and Jiang, Daxin and Majumder, Rangan and Wei, Furu},
journal={arXiv preprint arXiv:2212.03533},
year={2022}
}
局限性
该模型仅适用于英文文本。长文本将被截断至最多 512 个 token。
正在翻译中，请稍候...