starlight-apk/Qwen
1
0
Fork 0
mirror of https://github.com/QwenLM/Qwen.git synced 2026-05-20 16:35:47 +08:00
Code Issues Packages Projects Releases Wiki Activity

update agent benchmarks and add qwen-72b results

Browse Source
This commit is contained in:
兼欣
2023-12-06 12:57:11 +08:00
parent a0a557aad8
commit 7eb9016908
7 changed files with 314 additions and 771 deletions
Download Patch File Download Diff File Expand all files Collapse all files

212
README.md
View File

@@ -1066,22 +1066,28 @@ We have tested the model's tool calling capabilities on our open-source Chinese
<table> <table>
<tr> <tr>
<th colspan="4" align="center">Chinese Tool-Use Benchmark</th> <th colspan="4" align="center">Chinese Tool-Use Benchmark (Version 20231206)</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th> <th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">95%</td><td align="center">0.90</td><td align="center">15.0%</td> <td>GPT-4</td><td align="center">98.0%</td><td align="center">0.953</td><td align="center">23.9%</td>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">85%</td><td align="center">0.88</td><td align="center">75.0%</td> <td>GPT-3.5</td><td align="center">74.5%</td><td align="center">0.807</td><td align="center">80.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td><td align="center">98%</td><td align="center">0.91</td><td align="center">7.3%</td> <td>Qwen-1_8B-Chat</td><td align="center">85.0%</td><td align="center">0.839</td><td align="center">27.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td><td align="center">98%</td><td align="center">0.93</td><td align="center">2.4%</td> <td>Qwen-7B-Chat</td><td align="center">95.5%</td><td align="center">0.900</td><td align="center">11.6%</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">96.9%</td><td align="center">0.917</td><td align="center">5.6%</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td><td align="center">98.2%</td><td align="center">0.927</td><td align="center">1.1%</td>
</tr> </tr>
</table> </table>
@@ -1091,127 +1097,85 @@ We have observed that Qwen performs well in terms of code executability and resu
<table> <table>
<tr> <tr>
<th colspan="4" align="center">Executable Rate of Generated Code (%)</th> <th colspan="5" align="center">Code Interpreter Benchmark (Version 20231206)</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization↑</th><th align="center">General↑</th> <th rowspan="2" align="center">Model</th>
<th colspan="3" align="center">Accuracy of Code Execution Results (%)</th>
<th colspan="1" align="center">Executable Rate of Code (%)</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">91.9</td><td align="center">85.9</td><td align="center">82.8</td> <th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th><th align="center">General↑</th>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">89.2</td><td align="center">65.0</td><td align="center">74.1</td> <td>GPT-4</td>
<td align="center">82.8</td>
<td align="center">66.7</td>
<td align="center">60.8</td>
<td align="center">82.8</td>
</tr> </tr>
<tr> <tr>
<td>LLaMA2-7B-Chat</td> <td>GPT-3.5</td>
<td align="center">41.9</td> <td align="center">47.3</td>
<td align="center">33.1</td> <td align="center">33.3</td>
<td align="center">24.1 </td> <td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-13B-Chat</td>
<td align="center">50.0</td>
<td align="center">40.5</td>
<td align="center">48.3 </td>
</tr>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">85.1</td>
<td align="center">54.0</td>
<td align="center">70.7 </td>
</tr>
<tr>
<td>CodeLLaMA-13B-Instruct</td>
<td align="center">93.2</td>
<td align="center">55.8</td>
<td align="center">74.1</td> <td align="center">74.1</td>
</tr> </tr>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">78.4</td>
<td align="center">44.2</td>
<td align="center">62.1 </td>
</tr>
<tr>
<td>InternLM-20B-Chat</td>
<td align="center">70.3</td>
<td align="center">44.2</td>
<td align="center">65.5 </td>
</tr>
<tr>
<td>Qwen-7B-Chat</td>
<td align="center">82.4</td>
<td align="center">64.4</td>
<td align="center">67.2 </td>
</tr>
<tr>
<td>Qwen-14B-Chat</td>
<td align="center">89.2</td>
<td align="center">84.1</td>
<td align="center">65.5</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">Accuracy of Code Execution Results (%)</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">82.8</td><td align="center">66.7</td><td align="center">60.8</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">47.3</td><td align="center">33.3</td><td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-7B-Chat</td>
<td align="center">3.9</td>
<td align="center">14.3</td>
<td align="center">39.2 </td>
</tr>
<tr> <tr>
<td>LLaMA2-13B-Chat</td> <td>LLaMA2-13B-Chat</td>
<td align="center">8.3</td> <td align="center">8.3</td>
<td align="center">8.3</td> <td align="center">1.2</td>
<td align="center">40.5 </td> <td align="center">15.2</td>
</tr> <td align="center">48.3</td>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">14.3</td>
<td align="center">26.2</td>
<td align="center">60.8 </td>
</tr> </tr>
<tr> <tr>
<td>CodeLLaMA-13B-Instruct</td> <td>CodeLLaMA-13B-Instruct</td>
<td align="center">28.2</td> <td align="center">28.2</td>
<td align="center">27.4</td> <td align="center">15.5</td>
<td align="center">62.0 </td> <td align="center">21.5</td>
</tr> <td align="center">74.1</td>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">28.5</td>
<td align="center">4.8</td>
<td align="center">40.5 </td>
</tr> </tr>
<tr> <tr>
<td>InternLM-20B-Chat</td> <td>InternLM-20B-Chat</td>
<td align="center">34.6</td> <td align="center">34.6</td>
<td align="center">10.7</td>
<td align="center">25.1</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>ChatGLM3-6B</td>
<td align="center">54.2</td>
<td align="center">15.5</td>
<td align="center">21.5</td>
<td align="center">67.1</td>
</tr>
<tr>
<td>Qwen-1.8B-Chat</td>
<td align="center">25.6</td>
<td align="center">21.4</td> <td align="center">21.4</td>
<td align="center">45.6 </td> <td align="center">22.8</td>
<td align="center">65.5</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td> <td>Qwen-7B-Chat</td>
<td align="center">41.9</td> <td align="center">41.9</td>
<td align="center">40.5</td> <td align="center">23.8</td>
<td align="center">54.4 </td> <td align="center">38.0</td>
<td align="center">67.2</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td> <td>Qwen-14B-Chat</td>
<td align="center">58.4</td> <td align="center">58.4</td>
<td align="center">53.6</td> <td align="center">31.0</td>
<td align="center">59.5</td> <td align="center">45.6</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td>
<td align="center">72.7</td>
<td align="center">41.7</td>
<td align="center">43.0</td>
<td align="center">82.8</td>
</tr> </tr>
</table> </table>
@@ -1221,62 +1185,6 @@ We have observed that Qwen performs well in terms of code executability and resu
<br> <br>
<p> <p>
In addition, we also provide experimental results demonstrating that our model is capable of acting as a HuggingFace Agent. For more information, please refer to the [example documentation](examples/transformers_agent.md). The model's performance on the evaluation dataset provided by Hugging Face is as follows:
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent Benchmark- Run Mode</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">100</td><td align="center">100</td><td align="center">97.4</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">95.4</td><td align="center">96.3</td><td align="center">87.0</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">86.1</td><td align="center">87.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">87.0</td><td align="center">88.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">87.0</td><td align="center">87.0</td><td align="center">71.5</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">93.5</td><td align="center">94.4</td><td align="center">87.0</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent Benchmark - Chat Mode</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">98.5</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">97.3</td><td align="center">96.8</td><td align="center">89.6</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">91.1</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">89.6</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">94.7</td><td align="center">94.7</td><td align="center">85.1</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">95.5</td>
</tr>
</table>
<br> <br>
## Long-Context Understanding ## Long-Context Understanding

212
README_CN.md
View File

@@ -1059,22 +1059,28 @@ Qwen-Chat针对工具使用、函数调用能力进行了优化。用户可以
<table> <table>
<tr> <tr>
<th colspan="4" align="center">中文工具调用评测基准</th> <th colspan="4" align="center">中文工具调用评测基准（版本 20231206）</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th> <th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">95%</td><td align="center">0.90</td><td align="center">15.0%</td> <td>GPT-4</td><td align="center">98.0%</td><td align="center">0.953</td><td align="center">23.9%</td>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">85%</td><td align="center">0.88</td><td align="center">75.0%</td> <td>GPT-3.5</td><td align="center">74.5%</td><td align="center">0.807</td><td align="center">80.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td><td align="center">98%</td><td align="center">0.91</td><td align="center">7.3%</td> <td>Qwen-1_8B-Chat</td><td align="center">85.0%</td><td align="center">0.839</td><td align="center">27.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td><td align="center">98%</td><td align="center">0.93</td><td align="center">2.4%</td> <td>Qwen-7B-Chat</td><td align="center">95.5%</td><td align="center">0.900</td><td align="center">11.6%</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">96.9%</td><td align="center">0.917</td><td align="center">5.6%</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td><td align="center">98.2%</td><td align="center">0.927</td><td align="center">1.1%</td>
</tr> </tr>
</table> </table>
@@ -1083,127 +1089,85 @@ Qwen-Chat针对工具使用、函数调用能力进行了优化。用户可以
<table> <table>
<tr> <tr>
<th colspan="4" align="center">生成代码的可执行率 (%)</th> <th colspan="5" align="center">Code Interpreter Benchmark (Version 20231206)</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization↑</th><th align="center">General↑</th> <th rowspan="2" align="center">Model</th>
<th colspan="3" align="center">代码执行结果正确性 (%)</th>
<th colspan="1" align="center">生成代码的可执行率 (%)</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">91.9</td><td align="center">85.9</td><td align="center">82.8</td> <th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th><th align="center">General↑</th>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">89.2</td><td align="center">65.0</td><td align="center">74.1</td> <td>GPT-4</td>
<td align="center">82.8</td>
<td align="center">66.7</td>
<td align="center">60.8</td>
<td align="center">82.8</td>
</tr> </tr>
<tr> <tr>
<td>LLaMA2-7B-Chat</td> <td>GPT-3.5</td>
<td align="center">41.9</td> <td align="center">47.3</td>
<td align="center">33.1</td> <td align="center">33.3</td>
<td align="center">24.1 </td> <td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-13B-Chat</td>
<td align="center">50.0</td>
<td align="center">40.5</td>
<td align="center">48.3 </td>
</tr>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">85.1</td>
<td align="center">54.0</td>
<td align="center">70.7 </td>
</tr>
<tr>
<td>CodeLLaMA-13B-Instruct</td>
<td align="center">93.2</td>
<td align="center">55.8</td>
<td align="center">74.1</td> <td align="center">74.1</td>
</tr> </tr>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">78.4</td>
<td align="center">44.2</td>
<td align="center">62.1 </td>
</tr>
<tr>
<td>InternLM-20B-Chat</td>
<td align="center">70.3</td>
<td align="center">44.2</td>
<td align="center">65.5 </td>
</tr>
<tr>
<td>Qwen-7B-Chat</td>
<td align="center">82.4</td>
<td align="center">64.4</td>
<td align="center">67.2 </td>
</tr>
<tr>
<td>Qwen-14B-Chat</td>
<td align="center">89.2</td>
<td align="center">84.1</td>
<td align="center">65.5</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">代码执行结果的正确率 (%)</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">82.8</td><td align="center">66.7</td><td align="center">60.8</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">47.3</td><td align="center">33.3</td><td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-7B-Chat</td>
<td align="center">3.9</td>
<td align="center">14.3</td>
<td align="center">39.2 </td>
</tr>
<tr> <tr>
<td>LLaMA2-13B-Chat</td> <td>LLaMA2-13B-Chat</td>
<td align="center">8.3</td> <td align="center">8.3</td>
<td align="center">8.3</td> <td align="center">1.2</td>
<td align="center">40.5 </td> <td align="center">15.2</td>
</tr> <td align="center">48.3</td>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">14.3</td>
<td align="center">26.2</td>
<td align="center">60.8 </td>
</tr> </tr>
<tr> <tr>
<td>CodeLLaMA-13B-Instruct</td> <td>CodeLLaMA-13B-Instruct</td>
<td align="center">28.2</td> <td align="center">28.2</td>
<td align="center">27.4</td> <td align="center">15.5</td>
<td align="center">62.0 </td> <td align="center">21.5</td>
</tr> <td align="center">74.1</td>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">28.5</td>
<td align="center">4.8</td>
<td align="center">40.5 </td>
</tr> </tr>
<tr> <tr>
<td>InternLM-20B-Chat</td> <td>InternLM-20B-Chat</td>
<td align="center">34.6</td> <td align="center">34.6</td>
<td align="center">10.7</td>
<td align="center">25.1</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>ChatGLM3-6B</td>
<td align="center">54.2</td>
<td align="center">15.5</td>
<td align="center">21.5</td>
<td align="center">67.1</td>
</tr>
<tr>
<td>Qwen-1.8B-Chat</td>
<td align="center">25.6</td>
<td align="center">21.4</td> <td align="center">21.4</td>
<td align="center">45.6 </td> <td align="center">22.8</td>
<td align="center">65.5</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td> <td>Qwen-7B-Chat</td>
<td align="center">41.9</td> <td align="center">41.9</td>
<td align="center">40.5</td> <td align="center">23.8</td>
<td align="center">54.4 </td> <td align="center">38.0</td>
<td align="center">67.2</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td> <td>Qwen-14B-Chat</td>
<td align="center">58.4</td> <td align="center">58.4</td>
<td align="center">53.6</td> <td align="center">31.0</td>
<td align="center">59.5</td> <td align="center">45.6</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td>
<td align="center">72.7</td>
<td align="center">41.7</td>
<td align="center">43.0</td>
<td align="center">82.8</td>
</tr> </tr>
</table> </table>
@@ -1213,62 +1177,6 @@ Qwen-Chat针对工具使用、函数调用能力进行了优化。用户可以
<br> <br>
<p> <p>
此外，我们还提供了实验结果表明我们的模型具备扮演HuggingFace Agent的能力，详见[示例文档](examples/transformers_agent.md)了解更多信息。模型在Hugging Face提供的评测数据集上表现如下：
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent评测基准 - Run模式</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">100</td><td align="center">100</td><td align="center">97.4</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">95.4</td><td align="center">96.3</td><td align="center">87.0</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">86.1</td><td align="center">87.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">87.0</td><td align="center">88.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">87.0</td><td align="center">87.0</td><td align="center">71.5</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">93.5</td><td align="center">94.4</td><td align="center">87.0</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent评测基准 - Chat模式</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">98.5</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">97.3</td><td align="center">96.8</td><td align="center">89.6</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">91.1</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">89.6</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">94.7</td><td align="center">94.7</td><td align="center">85.1</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">95.5</td>
</tr>
</table>
<br> <br>
## 长文本理解 ## 长文本理解

212
README_ES.md
View File

@@ -1026,22 +1026,28 @@ Hemos probado las capacidades de llamada de la herramienta del modelo en nuestro
<table> <table>
<tr> <tr>
<th colspan="4" align="center">Chinese Tool-Use Benchmark</th> <th colspan="4" align="center">Chinese Tool-Use Benchmark (Version 20231206)</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th> <th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">95%</td><td align="center">0.90</td><td align="center">15.0%</td> <td>GPT-4</td><td align="center">98.0%</td><td align="center">0.953</td><td align="center">23.9%</td>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">85%</td><td align="center">0.88</td><td align="center">75.0%</td> <td>GPT-3.5</td><td align="center">74.5%</td><td align="center">0.807</td><td align="center">80.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td><td align="center">98%</td><td align="center">0.91</td><td align="center">7.3%</td> <td>Qwen-1_8B-Chat</td><td align="center">85.0%</td><td align="center">0.839</td><td align="center">27.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td><td align="center">98%</td><td align="center">0.93</td><td align="center">2.4%</td> <td>Qwen-7B-Chat</td><td align="center">95.5%</td><td align="center">0.900</td><td align="center">11.6%</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">96.9%</td><td align="center">0.917</td><td align="center">5.6%</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td><td align="center">98.2%</td><td align="center">0.927</td><td align="center">1.1%</td>
</tr> </tr>
</table> </table>
@@ -1051,127 +1057,85 @@ Hemos observado que Qwen funciona bien en términos de ejecutabilidad del códig
<table> <table>
<tr> <tr>
<th colspan="4" align="center">Executable Rate of Generated Code (%)</th> <th colspan="5" align="center">Code Interpreter Benchmark (Version 20231206)</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization↑</th><th align="center">General↑</th> <th rowspan="2" align="center">Model</th>
<th colspan="3" align="center">Accuracy of Code Execution Results (%)</th>
<th colspan="1" align="center">Executable Rate of Code (%)</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">91.9</td><td align="center">85.9</td><td align="center">82.8</td> <th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th><th align="center">General↑</th>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">89.2</td><td align="center">65.0</td><td align="center">74.1</td> <td>GPT-4</td>
<td align="center">82.8</td>
<td align="center">66.7</td>
<td align="center">60.8</td>
<td align="center">82.8</td>
</tr> </tr>
<tr> <tr>
<td>LLaMA2-7B-Chat</td> <td>GPT-3.5</td>
<td align="center">41.9</td> <td align="center">47.3</td>
<td align="center">33.1</td> <td align="center">33.3</td>
<td align="center">24.1 </td> <td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-13B-Chat</td>
<td align="center">50.0</td>
<td align="center">40.5</td>
<td align="center">48.3 </td>
</tr>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">85.1</td>
<td align="center">54.0</td>
<td align="center">70.7 </td>
</tr>
<tr>
<td>CodeLLaMA-13B-Instruct</td>
<td align="center">93.2</td>
<td align="center">55.8</td>
<td align="center">74.1</td> <td align="center">74.1</td>
</tr> </tr>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">78.4</td>
<td align="center">44.2</td>
<td align="center">62.1 </td>
</tr>
<tr>
<td>InternLM-20B-Chat</td>
<td align="center">70.3</td>
<td align="center">44.2</td>
<td align="center">65.5 </td>
</tr>
<tr>
<td>Qwen-7B-Chat</td>
<td align="center">82.4</td>
<td align="center">64.4</td>
<td align="center">67.2 </td>
</tr>
<tr>
<td>Qwen-14B-Chat</td>
<td align="center">89.2</td>
<td align="center">84.1</td>
<td align="center">65.5</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">Accuracy of Code Execution Results (%)</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">82.8</td><td align="center">66.7</td><td align="center">60.8</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">47.3</td><td align="center">33.3</td><td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-7B-Chat</td>
<td align="center">3.9</td>
<td align="center">14.3</td>
<td align="center">39.2 </td>
</tr>
<tr> <tr>
<td>LLaMA2-13B-Chat</td> <td>LLaMA2-13B-Chat</td>
<td align="center">8.3</td> <td align="center">8.3</td>
<td align="center">8.3</td> <td align="center">1.2</td>
<td align="center">40.5 </td> <td align="center">15.2</td>
</tr> <td align="center">48.3</td>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">14.3</td>
<td align="center">26.2</td>
<td align="center">60.8 </td>
</tr> </tr>
<tr> <tr>
<td>CodeLLaMA-13B-Instruct</td> <td>CodeLLaMA-13B-Instruct</td>
<td align="center">28.2</td> <td align="center">28.2</td>
<td align="center">27.4</td> <td align="center">15.5</td>
<td align="center">62.0 </td> <td align="center">21.5</td>
</tr> <td align="center">74.1</td>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">28.5</td>
<td align="center">4.8</td>
<td align="center">40.5 </td>
</tr> </tr>
<tr> <tr>
<td>InternLM-20B-Chat</td> <td>InternLM-20B-Chat</td>
<td align="center">34.6</td> <td align="center">34.6</td>
<td align="center">10.7</td>
<td align="center">25.1</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>ChatGLM3-6B</td>
<td align="center">54.2</td>
<td align="center">15.5</td>
<td align="center">21.5</td>
<td align="center">67.1</td>
</tr>
<tr>
<td>Qwen-1.8B-Chat</td>
<td align="center">25.6</td>
<td align="center">21.4</td> <td align="center">21.4</td>
<td align="center">45.6 </td> <td align="center">22.8</td>
<td align="center">65.5</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td> <td>Qwen-7B-Chat</td>
<td align="center">41.9</td> <td align="center">41.9</td>
<td align="center">40.5</td> <td align="center">23.8</td>
<td align="center">54.4 </td> <td align="center">38.0</td>
<td align="center">67.2</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td> <td>Qwen-14B-Chat</td>
<td align="center">58.4</td> <td align="center">58.4</td>
<td align="center">53.6</td> <td align="center">31.0</td>
<td align="center">59.5</td> <td align="center">45.6</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td>
<td align="center">72.7</td>
<td align="center">41.7</td>
<td align="center">43.0</td>
<td align="center">82.8</td>
</tr> </tr>
</table> </table>
@@ -1181,62 +1145,6 @@ Hemos observado que Qwen funciona bien en términos de ejecutabilidad del códig
<br> <br>
<p> <p>
Además, también proporcionamos resultados experimentales que demuestran que nuestro modelo es capaz de actuar como un Agente HuggingFace. Para más información, consulte la [documentación del ejemplo](examples/transformers_agent.md). El rendimiento del modelo en el conjunto de datos de evaluación proporcionado por Hugging Face es el siguiente:
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent Benchmark- Run Mode</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">100</td><td align="center">100</td><td align="center">97.4</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">95.4</td><td align="center">96.3</td><td align="center">87.0</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">86.1</td><td align="center">87.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">87.0</td><td align="center">88.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">87.0</td><td align="center">87.0</td><td align="center">71.5</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">93.5</td><td align="center">94.4</td><td align="center">87.0</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent Benchmark - Chat Mode</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">98.5</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">97.3</td><td align="center">96.8</td><td align="center">89.6</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">91.1</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">89.6</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">94.7</td><td align="center">94.7</td><td align="center">85.1</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">95.5</td>
</tr>
</table>
<br> <br>
## Comprensión del Contexto Largo ## Comprensión del Contexto Largo

212
README_FR.md
View File

@@ -1029,22 +1029,28 @@ Nous avons testé les capacités d'appel d'outil du modèle sur notre benchmark
<table> <table>
<tr> <tr>
<th colspan="4" align="center">Chinese Tool-Use Benchmark</th> <th colspan="4" align="center">Chinese Tool-Use Benchmark (Version 20231206)</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th> <th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">95%</td><td align="center">0.90</td><td align="center">15.0%</td> <td>GPT-4</td><td align="center">98.0%</td><td align="center">0.953</td><td align="center">23.9%</td>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">85%</td><td align="center">0.88</td><td align="center">75.0%</td> <td>GPT-3.5</td><td align="center">74.5%</td><td align="center">0.807</td><td align="center">80.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td><td align="center">98%</td><td align="center">0.91</td><td align="center">7.3%</td> <td>Qwen-1_8B-Chat</td><td align="center">85.0%</td><td align="center">0.839</td><td align="center">27.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td><td align="center">98%</td><td align="center">0.93</td><td align="center">2.4%</td> <td>Qwen-7B-Chat</td><td align="center">95.5%</td><td align="center">0.900</td><td align="center">11.6%</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">96.9%</td><td align="center">0.917</td><td align="center">5.6%</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td><td align="center">98.2%</td><td align="center">0.927</td><td align="center">1.1%</td>
</tr> </tr>
</table> </table>
@@ -1054,127 +1060,85 @@ Nous avons observé que Qwen est performant en termes d'exécutabilité du code
<table> <table>
<tr> <tr>
<th colspan="4" align="center">Executable Rate of Generated Code (%)</th> <th colspan="5" align="center">Code Interpreter Benchmark (Version 20231206)</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization↑</th><th align="center">General↑</th> <th rowspan="2" align="center">Model</th>
<th colspan="3" align="center">Accuracy of Code Execution Results (%)</th>
<th colspan="1" align="center">Executable Rate of Code (%)</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">91.9</td><td align="center">85.9</td><td align="center">82.8</td> <th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th><th align="center">General↑</th>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">89.2</td><td align="center">65.0</td><td align="center">74.1</td> <td>GPT-4</td>
<td align="center">82.8</td>
<td align="center">66.7</td>
<td align="center">60.8</td>
<td align="center">82.8</td>
</tr> </tr>
<tr> <tr>
<td>LLaMA2-7B-Chat</td> <td>GPT-3.5</td>
<td align="center">41.9</td> <td align="center">47.3</td>
<td align="center">33.1</td> <td align="center">33.3</td>
<td align="center">24.1 </td> <td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-13B-Chat</td>
<td align="center">50.0</td>
<td align="center">40.5</td>
<td align="center">48.3 </td>
</tr>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">85.1</td>
<td align="center">54.0</td>
<td align="center">70.7 </td>
</tr>
<tr>
<td>CodeLLaMA-13B-Instruct</td>
<td align="center">93.2</td>
<td align="center">55.8</td>
<td align="center">74.1</td> <td align="center">74.1</td>
</tr> </tr>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">78.4</td>
<td align="center">44.2</td>
<td align="center">62.1 </td>
</tr>
<tr>
<td>InternLM-20B-Chat</td>
<td align="center">70.3</td>
<td align="center">44.2</td>
<td align="center">65.5 </td>
</tr>
<tr>
<td>Qwen-7B-Chat</td>
<td align="center">82.4</td>
<td align="center">64.4</td>
<td align="center">67.2 </td>
</tr>
<tr>
<td>Qwen-14B-Chat</td>
<td align="center">89.2</td>
<td align="center">84.1</td>
<td align="center">65.5</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">Accuracy of Code Execution Results (%)</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">82.8</td><td align="center">66.7</td><td align="center">60.8</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">47.3</td><td align="center">33.3</td><td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-7B-Chat</td>
<td align="center">3.9</td>
<td align="center">14.3</td>
<td align="center">39.2 </td>
</tr>
<tr> <tr>
<td>LLaMA2-13B-Chat</td> <td>LLaMA2-13B-Chat</td>
<td align="center">8.3</td> <td align="center">8.3</td>
<td align="center">8.3</td> <td align="center">1.2</td>
<td align="center">40.5 </td> <td align="center">15.2</td>
</tr> <td align="center">48.3</td>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">14.3</td>
<td align="center">26.2</td>
<td align="center">60.8 </td>
</tr> </tr>
<tr> <tr>
<td>CodeLLaMA-13B-Instruct</td> <td>CodeLLaMA-13B-Instruct</td>
<td align="center">28.2</td> <td align="center">28.2</td>
<td align="center">27.4</td> <td align="center">15.5</td>
<td align="center">62.0 </td> <td align="center">21.5</td>
</tr> <td align="center">74.1</td>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">28.5</td>
<td align="center">4.8</td>
<td align="center">40.5 </td>
</tr> </tr>
<tr> <tr>
<td>InternLM-20B-Chat</td> <td>InternLM-20B-Chat</td>
<td align="center">34.6</td> <td align="center">34.6</td>
<td align="center">10.7</td>
<td align="center">25.1</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>ChatGLM3-6B</td>
<td align="center">54.2</td>
<td align="center">15.5</td>
<td align="center">21.5</td>
<td align="center">67.1</td>
</tr>
<tr>
<td>Qwen-1.8B-Chat</td>
<td align="center">25.6</td>
<td align="center">21.4</td> <td align="center">21.4</td>
<td align="center">45.6 </td> <td align="center">22.8</td>
<td align="center">65.5</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td> <td>Qwen-7B-Chat</td>
<td align="center">41.9</td> <td align="center">41.9</td>
<td align="center">40.5</td> <td align="center">23.8</td>
<td align="center">54.4 </td> <td align="center">38.0</td>
<td align="center">67.2</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td> <td>Qwen-14B-Chat</td>
<td align="center">58.4</td> <td align="center">58.4</td>
<td align="center">53.6</td> <td align="center">31.0</td>
<td align="center">59.5</td> <td align="center">45.6</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td>
<td align="center">72.7</td>
<td align="center">41.7</td>
<td align="center">43.0</td>
<td align="center">82.8</td>
</tr> </tr>
</table> </table>
@@ -1184,62 +1148,6 @@ Nous avons observé que Qwen est performant en termes d'exécutabilité du code
<br> <br>
<p> <p>
En outre, nous fournissons également des résultats expérimentaux démontrant que notre modèle est capable d'agir en tant qu'agent Hugging Face. Pour plus d'informations, veuillez vous référer à la [documentation de l'exemple](examples/transformers_agent.md). Les performances du modèle sur l'ensemble des données d'évaluation fournies par Hugging Face sont les suivantes:
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent Benchmark- Run Mode</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">100</td><td align="center">100</td><td align="center">97.4</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">95.4</td><td align="center">96.3</td><td align="center">87.0</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">86.1</td><td align="center">87.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">87.0</td><td align="center">88.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">87.0</td><td align="center">87.0</td><td align="center">71.5</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">93.5</td><td align="center">94.4</td><td align="center">87.0</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent Benchmark - Chat Mode</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">98.5</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">97.3</td><td align="center">96.8</td><td align="center">89.6</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">91.1</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">89.6</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">94.7</td><td align="center">94.7</td><td align="center">85.1</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">95.5</td>
</tr>
</table>
<br> <br>
## Compréhension du Contexte Long ## Compréhension du Contexte Long

212
README_JA.md
View File

@@ -1056,22 +1056,28 @@ ReAct プロンプトの原則に基づいてツール呼び出しを実装す
<table> <table>
<tr> <tr>
<th colspan="4" align="center">Chinese Tool-Use Benchmark</th> <th colspan="4" align="center">Chinese Tool-Use Benchmark (Version 20231206)</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th> <th align="center">Model</th><th align="center">Tool Selection (Acc.↑)</th><th align="center">Tool Input (Rouge-L↑)</th><th align="center">False Positive Error↓</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">95%</td><td align="center">0.90</td><td align="center">15.0%</td> <td>GPT-4</td><td align="center">98.0%</td><td align="center">0.953</td><td align="center">23.9%</td>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">85%</td><td align="center">0.88</td><td align="center">75.0%</td> <td>GPT-3.5</td><td align="center">74.5%</td><td align="center">0.807</td><td align="center">80.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td><td align="center">98%</td><td align="center">0.91</td><td align="center">7.3%</td> <td>Qwen-1_8B-Chat</td><td align="center">85.0%</td><td align="center">0.839</td><td align="center">27.6%</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td><td align="center">98%</td><td align="center">0.93</td><td align="center">2.4%</td> <td>Qwen-7B-Chat</td><td align="center">95.5%</td><td align="center">0.900</td><td align="center">11.6%</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">96.9%</td><td align="center">0.917</td><td align="center">5.6%</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td><td align="center">98.2%</td><td align="center">0.927</td><td align="center">1.1%</td>
</tr> </tr>
</table> </table>
@@ -1081,127 +1087,85 @@ Qwen は、コード生成時のコードの実行可能性と結果の精度の
<table> <table>
<tr> <tr>
<th colspan="4" align="center">Executable Rate of Generated Code (%)</th> <th colspan="5" align="center">Code Interpreter Benchmark (Version 20231206)</th>
</tr> </tr>
<tr> <tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization↑</th><th align="center">General↑</th> <th rowspan="2" align="center">Model</th>
<th colspan="3" align="center">Accuracy of Code Execution Results (%)</th>
<th colspan="1" align="center">Executable Rate of Code (%)</th>
</tr> </tr>
<tr> <tr>
<td>GPT-4</td><td align="center">91.9</td><td align="center">85.9</td><td align="center">82.8</td> <th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th><th align="center">General↑</th>
</tr> </tr>
<tr> <tr>
<td>GPT-3.5</td><td align="center">89.2</td><td align="center">65.0</td><td align="center">74.1</td> <td>GPT-4</td>
<td align="center">82.8</td>
<td align="center">66.7</td>
<td align="center">60.8</td>
<td align="center">82.8</td>
</tr> </tr>
<tr> <tr>
<td>LLaMA2-7B-Chat</td> <td>GPT-3.5</td>
<td align="center">41.9</td> <td align="center">47.3</td>
<td align="center">33.1</td> <td align="center">33.3</td>
<td align="center">24.1 </td> <td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-13B-Chat</td>
<td align="center">50.0</td>
<td align="center">40.5</td>
<td align="center">48.3 </td>
</tr>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">85.1</td>
<td align="center">54.0</td>
<td align="center">70.7 </td>
</tr>
<tr>
<td>CodeLLaMA-13B-Instruct</td>
<td align="center">93.2</td>
<td align="center">55.8</td>
<td align="center">74.1</td> <td align="center">74.1</td>
</tr> </tr>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">78.4</td>
<td align="center">44.2</td>
<td align="center">62.1 </td>
</tr>
<tr>
<td>InternLM-20B-Chat</td>
<td align="center">70.3</td>
<td align="center">44.2</td>
<td align="center">65.5 </td>
</tr>
<tr>
<td>Qwen-7B-Chat</td>
<td align="center">82.4</td>
<td align="center">64.4</td>
<td align="center">67.2 </td>
</tr>
<tr>
<td>Qwen-14B-Chat</td>
<td align="center">89.2</td>
<td align="center">84.1</td>
<td align="center">65.5</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">Accuracy of Code Execution Results (%)</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">82.8</td><td align="center">66.7</td><td align="center">60.8</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">47.3</td><td align="center">33.3</td><td align="center">55.7</td>
</tr>
<tr>
<td>LLaMA2-7B-Chat</td>
<td align="center">3.9</td>
<td align="center">14.3</td>
<td align="center">39.2 </td>
</tr>
<tr> <tr>
<td>LLaMA2-13B-Chat</td> <td>LLaMA2-13B-Chat</td>
<td align="center">8.3</td> <td align="center">8.3</td>
<td align="center">8.3</td> <td align="center">1.2</td>
<td align="center">40.5 </td> <td align="center">15.2</td>
</tr> <td align="center">48.3</td>
<tr>
<td>CodeLLaMA-7B-Instruct</td>
<td align="center">14.3</td>
<td align="center">26.2</td>
<td align="center">60.8 </td>
</tr> </tr>
<tr> <tr>
<td>CodeLLaMA-13B-Instruct</td> <td>CodeLLaMA-13B-Instruct</td>
<td align="center">28.2</td> <td align="center">28.2</td>
<td align="center">27.4</td> <td align="center">15.5</td>
<td align="center">62.0 </td> <td align="center">21.5</td>
</tr> <td align="center">74.1</td>
<tr>
<td>InternLM-7B-Chat-v1.1</td>
<td align="center">28.5</td>
<td align="center">4.8</td>
<td align="center">40.5 </td>
</tr> </tr>
<tr> <tr>
<td>InternLM-20B-Chat</td> <td>InternLM-20B-Chat</td>
<td align="center">34.6</td> <td align="center">34.6</td>
<td align="center">10.7</td>
<td align="center">25.1</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>ChatGLM3-6B</td>
<td align="center">54.2</td>
<td align="center">15.5</td>
<td align="center">21.5</td>
<td align="center">67.1</td>
</tr>
<tr>
<td>Qwen-1.8B-Chat</td>
<td align="center">25.6</td>
<td align="center">21.4</td> <td align="center">21.4</td>
<td align="center">45.6 </td> <td align="center">22.8</td>
<td align="center">65.5</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-7B-Chat</td> <td>Qwen-7B-Chat</td>
<td align="center">41.9</td> <td align="center">41.9</td>
<td align="center">40.5</td> <td align="center">23.8</td>
<td align="center">54.4 </td> <td align="center">38.0</td>
<td align="center">67.2</td>
</tr> </tr>
<tr> <tr>
<td>Qwen-14B-Chat</td> <td>Qwen-14B-Chat</td>
<td align="center">58.4</td> <td align="center">58.4</td>
<td align="center">53.6</td> <td align="center">31.0</td>
<td align="center">59.5</td> <td align="center">45.6</td>
<td align="center">65.5</td>
</tr>
<tr>
<td>Qwen-72B-Chat</td>
<td align="center">72.7</td>
<td align="center">41.7</td>
<td align="center">43.0</td>
<td align="center">82.8</td>
</tr> </tr>
</table> </table>
@@ -1211,62 +1175,6 @@ Qwen は、コード生成時のコードの実行可能性と結果の精度の
<br> <br>
<p> <p>
さらに、Qwenが HuggingFace Agent として機能できることを実証する実験結果も提供します。 詳細については、[ドキュメント例](examples/transformers_agent.md) を参照してください。 Hugging Face が提供する評価データセットにおけるモデルのパフォーマンスは次のとおりです。
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent Benchmark- Run Mode</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">100</td><td align="center">100</td><td align="center">97.4</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">95.4</td><td align="center">96.3</td><td align="center">87.0</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">86.1</td><td align="center">87.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">87.0</td><td align="center">88.0</td><td align="center">68.9</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">87.0</td><td align="center">87.0</td><td align="center">71.5</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">93.5</td><td align="center">94.4</td><td align="center">87.0</td>
</tr>
</table>
<table>
<tr>
<th colspan="4" align="center">HuggingFace Agent Benchmark - Chat Mode</th>
</tr>
<tr>
<th align="center">Model</th><th align="center">Tool Selection↑</th><th align="center">Tool Used↑</th><th align="center">Code↑</th>
</tr>
<tr>
<td>GPT-4</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">98.5</td>
</tr>
<tr>
<td>GPT-3.5</td><td align="center">97.3</td><td align="center">96.8</td><td align="center">89.6</td>
</tr>
<tr>
<td>StarCoder-Base-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">91.1</td>
</tr>
<tr>
<td>StarCoder-15B</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">89.6</td>
</tr>
<tr>
<td>Qwen-7B-Chat</td><td align="center">94.7</td><td align="center">94.7</td><td align="center">85.1</td>
</tr>
<tr>
<td>Qwen-14B-Chat</td><td align="center">97.9</td><td align="center">97.9</td><td align="center">95.5</td>
</tr>
</table>
<br> <br>
## 長い文脈の理解 ## 長い文脈の理解

9
eval/EVALUATION.md
View File

@@ -85,9 +85,12 @@ This script is used to reproduce the results of the ReAct and Hugging Face Agent
# Qwen-7B-Chat # Qwen-7B-Chat
mkdir data; mkdir data;
cd data; cd data;
wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v1/exam_plugin_v1_react_positive.jsonl; ## Old Evaluation Dataset (Version 20230803)
wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v1/exam_plugin_v1_react_negative.jsonl; # wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v1/exam_plugin_v1_react_positive.jsonl;
cd ..; # wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v1/exam_plugin_v1_react_negative.jsonl;
## New Evaluation Dataset (Version 20231206)
wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v20231206/exam_plugin_v20231206_react_positive.jsonl;
wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/opensource_data/exam_plugin_v20231206/exam_plugin_v20231206_react_negative.jsonl;cd ..;
pip install json5; pip install json5;
pip install jsonlines; pip install jsonlines;
pip install rouge_score; pip install rouge_score;

4
eval/evaluate_plugin.py
View File

@@ -46,7 +46,7 @@ def process_res(response):
) )
except: except:
# print("JSON Load Error:", action_input) # print("JSON Load Error:", action_input)
pass action_input = ""
res_dict = { res_dict = {
"thought": thought, "thought": thought,
"action": action, "action": action,
@@ -80,7 +80,7 @@ def eval_action(job):
response = job["gen"][0] response = job["gen"][0]
golden = job["response"] golden = job["response"]
if "Action:" in response: if "\nAction: " in response:
response, golden = process_res(response), process_res(golden) response, golden = process_res(response), process_res(golden)
if is_callable(response, golden): if is_callable(response, golden):
return True return True