starlight-apk/Qwen
1
0
Fork 0
mirror of https://github.com/QwenLM/Qwen.git synced 2026-05-20 08:25:47 +08:00
Code Issues Packages Projects Releases Wiki Activity

init commit of recipes (#1027)

Browse Source 
Add recipes
This commit is contained in:
chywang
2024-01-30 01:57:09 -06:00
committed by GitHub
parent d275e5b91a
commit ee01f36ed9
30 changed files with 5146 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

131
recipes/applications/chatbot/qwen_chatbot.ipynb Normal file
View File

@@ -0,0 +1,131 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "54d5d255-aa98-4655-8dd1-bc726430d86a",
"metadata": {},
"source": [
"# Qwen-7B-Chat Chat Demo"
]
},
{
"cell_type": "markdown",
"id": "31e04af4-eb27-4802-a7b2-6ea0525f1dc8",
"metadata": {},
"source": [
"This notebook uses Qwen-7B-Chat as an example to introduce you to how to build a web-based conversational assistant using Gradio."
]
},
{
"cell_type": "markdown",
"id": "75e51155-9f8e-40dc-8432-60f4567d93a8",
"metadata": {},
"source": [
"## Preparation"
]
},
{
"cell_type": "markdown",
"id": "ff6f061c-a033-49f2-8f7d-af3f23ac9125",
"metadata": {},
"source": [
"Download Qwen-7B-Chat\n",
"\n",
"Firstly, we need to download the model. You can use the snapshot_download that comes with modelscope to download the model to a specified directory."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c469a129-451f-4d01-8bc0-e2cf70a262c8",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!pip install modelscope"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "69af626e-22b8-49ad-8869-8354f4c72bcc",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"from modelscope.hub.snapshot_download import snapshot_download\n",
"snapshot_download(\"qwen/Qwen-7B-Chat\",cache_dir='/tmp/models') "
]
},
{
"cell_type": "markdown",
"id": "01d2ff34-4053-4710-a289-e354673be1ca",
"metadata": {},
"source": [
"## Install Dependencies"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "48b51791-4bbc-4d12-9cd6-587c24c8bea7",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!pip install -r ../../../requirements.txt\n",
"!pip install gradio==3.37.0 mdtex2html"
]
},
{
"cell_type": "markdown",
"id": "7732037a-246a-4953-af07-dae7a3ae5937",
"metadata": {},
"source": [
"## Run the web UI code to start the Qwen chatbot\n",
"\n",
"Users can run the web_demo.py file to have real-time conversations with Qwen-7b-chat on the webpage."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9e256f0a-d96d-4fd7-b305-fe43c6959dc8",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"tags": []
},
"outputs": [],
"source": [
"!python ../../../web_demo.py -c /tmp/models/qwen/Qwen-7B-Chat"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

433
recipes/applications/domain_finetune/qwen_domain_finetune.ipynb Normal file
View File

@@ -0,0 +1,433 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "0e7993c3-3999-4ac5-b1dc-77875d80e4c8",
"metadata": {},
"source": [
"# Fine-tuning Qwen-7B-Chat on Your Own Domain-Specific Data\n",
"\n",
"This notebook uses Qwen-7B-Chat as an example to introduce how to LoRA fine-tune the Qwen model on a specific domain.\n",
"\n",
"Qwen is a pretrained conversational model that supports English and Chinese. It is suitable for universal scenarios, but may lack some specialized knowledge in certain specific domain. If you would like to fine-tune it for a specific domain, or on your own private dataset, you can refer to this tutorial.\n",
"\n",
"Here is an example showing the differences before and after fine-tuning.\n",
"\n",
"#### Model Input\n",
">怎样治疗青光眼?\n",
"\n",
"#### Before Fine-tuning\n",
"\n",
">青光眼的治疗方法取决于病情严重程度。轻度青光眼可以采取药物治疗，比如激素类药物、抗生素等，以减轻炎症和改善血管紧张。\n",
">\n",
">中度青光眼可能需要激光手术来降低眼压，并使用药物以保持控制。\n",
">\n",
">重度青光眼则需要实施传统的眼部手术，如小梁切除术或巩膜外减压术，以防止视神经损伤。\n",
"\n",
"#### After Fine-tuning\n",
"\n",
">青光眼是一种眼睛疾病，可以引起视神经的损伤和视力下降。治疗青光眼的主要方法是药物治疗、激光治疗和手术治疗。\n",
">1. 药物治疗：对于早期青光眼患者，通常采用抗青光眼药物治疗，通过降低眼压来预防视神经进一步损伤。目前常用的抗青光眼药物有拉坦前列素滴眼液、阿托品滴眼液等。\n",
">2. 激光治疗：对于中晚期青光眼患者，如果药物治疗效果不佳，可能需要进行激光治疗。目前常用的激光治疗方法包括小梁切除术、YAG激光手术等。\n",
">3. 手术治疗：对于非常严重的青光眼患者，可能需要进行手术治疗。目前常用的手术治疗方法包括前房引流术、虹膜周切术、小梁切除术、巩膜滤过术等。\n",
">\n",
">需要注意的是，以上治疗方法的具体选择应根据患者的病情和医生的专业建议来进行。同时，青光眼患者在日常生活中也需要保持良好的生活习惯，避免过度疲劳、保持情绪稳定、定期检查眼睛等情况的发生。"
]
},
{
"cell_type": "markdown",
"id": "bdea7e21-fec8-49fe-b7ea-afde3f02738f",
"metadata": {},
"source": [
"## Environment Requirements\n",
"\n",
"Please refer to **requirements.txt** to install the required dependencies.\n",
"\n",
"Run the following command line in the main directory of the Qwen repo.\n",
"```bash\n",
"pip install -r requirements.txt\n",
"```\n",
"\n",
"\n",
"## Preparation\n",
"\n",
"### Download Qwen-7B-Chat\n",
"\n",
"First, download the model files. You can choose to download directly from ModelScope."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "248488f9-4a86-4f35-9d56-50f8e91a8f11",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"tags": []
},
"outputs": [],
"source": [
"from modelscope.hub.snapshot_download import snapshot_download\n",
"model_dir = snapshot_download('Qwen/Qwen-7B-chat', cache_dir='.')"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7b2a92b1-f08e-4413-9f92-8f23761e6e1f",
"metadata": {},
"source": [
"### Download Medical Training Data\n",
"\n",
"Download the data required for training; here, we provide a medical conversation dataset for training. It is sampled from [MedicalGPT repo](https://github.com/shibing624/MedicalGPT/) and we have converted this dataset into a format that can be used for fine-tuning.\n",
"\n",
"Disclaimer: the dataset can be only used for the research purpose."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ce195f08-fbb2-470e-b6c0-9a03457458c7",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"tags": []
},
"outputs": [],
"source": [
"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/medical_sft.json"
]
},
{
"cell_type": "markdown",
"id": "7226bed0-171b-4d45-a3f9-b3d81ec2bb9f",
"metadata": {},
"source": [
"You can prepare your dataset in JSON format following the format below, and then modify the `--data_path` parameter in the training command to point to your JSON file.\n",
"\n",
"These data instances can be conversations in the real world or include domain knowledge QA pairs. Besides, fine-tuning allows Qwen-chat to play like some specific roles. As Qwen-chat is a dialogue model for general scenarios, your fine-tuning can customize a chatbot to meet your requirements.\n",
"\n",
"We recommend that you prepare 50~ data instances if you want to fine-tune Qwen-chat as a roleplay model.\n",
"\n",
"You may prepare much more data instances if you want to infuse the domain knowledge of your field into the model.\n",
"\n",
"In this tutorial, we have prepared a medical domain fine-tuning dataset consisting of 1000 data instancess as an example. You can refer to our example to fine-tune on your own domain-specific dataset.\n",
"\n",
"Below is a simple example list with 1 sample:\n",
"\n",
"```python\n",
"[\n",
" {\n",
" \"id\": \"1\",\n",
" \"conversations\": [\n",
" {\n",
" \"from\": \"user\",\n",
" \"value\": \"直肠腺瘤的早期症状?\"\n",
" },\n",
" {\n",
" \"from\": \"assistant\",\n",
" \"value\": \"结直肠腺瘤是起源于结直肠黏膜腺上皮的良性肿瘤，包括结肠腺瘤与直肠腺瘤，是常见的肠道良性肿瘤。因与大肠癌的发生关系密切，被认为是一种癌前病变。不同地区、不同年龄的发病率差别很大，40岁以下的发病率低，60岁以上较高，男女无明显差别。此类疾病的病因及发病机制不明确。部分患者有遗传因素存在，另外，高脂肪饮食与食物纤维不足、肠道菌群紊乱也与此病的发生有相关性。多数患者无症状。症状与腺瘤的大小、部位、病理性质等有关。有症状者可有以下表现：直肠腺瘤的临床常见症状有便血，多为无疼性便血，常呈鲜红色，量多，可引起贫血。粪便性状改变，多为粘液便，便意不尽或里急重感，有时只排出粘液，称为假性腹泻。1.腹部不适、腹部疼痛，多为大的腺瘤伴发肠套叠、肠梗阻引起。2.排便习惯改变，包括便秘、腹泻、里急后重等。3.粪便带血，最常见为间歇性便血。4.部分位于直肠的较大的、带蒂腺瘤可在排便时脱落或脱出肛。可以采取内镜下高频电凝、激光、微波凝固等方法切除，也可以选择外科手术切除，并定期随访。有恶变者根据情况选择其他治疗（如放疗、化疗、手术等）管状腺瘤切除术后复发者少见，但绒毛状腺瘤及绒毛管状腺瘤切除术后常可复发，尤其是绒毛状腺瘤，且多发的腺瘤复发率高于单发者。对于经内镜治疗或局部手术切除的结直肠腺瘤患者尤其是绒毛状腺瘤或广基的绒毛管状腺瘤患者，建议腺瘤切除后的第一年内3～6个月进行一次肠镜检查，第二年开始每年一次。\"\n",
" }\n",
" ]\n",
" }\n",
"]\n",
"```\n",
"\n",
"You can also use multi-turn conversations as the training set. Here is a simple example:\n",
"\n",
"```python\n",
"[\n",
" {\n",
" \"id\": \"2\",\n",
" \"conversations\": [\n",
" {\n",
" \"from\": \"user\",\n",
" \"value\": \"你好，我最近经常感觉胸口疼痛，这是怎么回事？\"\n",
" },\n",
" {\n",
" \"from\": \"assistant\",\n",
" \"value\": \"胸痛可能由多种原因引起，包括心脏问题、消化系统疾病、呼吸系统问题等。您能描述一下疼痛的性质和持续时间吗？\"\n",
" },\n",
" {\n",
" \"from\": \"user\",\n",
" \"value\": \"疼痛感觉像是挤压，大概持续了几分钟。\"\n",
" },\n",
" {\n",
" \"from\": \"assistant\",\n",
" \"value\": \"挤压感的胸痛可能与心脏问题有关，特别是如果它伴随着呼吸困难、出汗或恶心。我建议您尽快去看医生并进行适当的检查，如心电图和血液检测，以确定具体原因。\"\n",
" },\n",
" {\n",
" \"from\": \"user\",\n",
" \"value\": \"我应该去急诊室吗，还是预约我的家庭医生？\"\n",
" },\n",
" {\n",
" \"from\": \"assistant\",\n",
" \"value\": \"如果您的疼痛是突然发生的，并且还有其他症状，建议您立即去急诊室。如果疼痛不是很严重且没有其他严重症状，您可以预约家庭医生进一步评估。但请不要忽视疼痛，尤其是如果这种情况是第一次出现。\"\n",
" }\n",
" ]\n",
" }\n",
"]\n",
"```\n",
"\n",
"## Fine-Tune the Model\n",
"\n",
"You can directly run the prepared training script to fine-tune the model. \n",
"\n",
"For parameter settings, you can modify `--model_name_or_path` to the location of the model you want to fine-tune, and set `--data_path` to the location of the dataset.\n",
"\n",
"You should remove the `--bf16` parameter if you are using a non-Ampere architecture GPU, such as a V100. \n",
"\n",
"For `--model_max_length` and `--per_device_train_batch_size`, we recommend the following configurations, ,you can refer to [this document](../../finetune/deepspeed/readme.md) for more details:\n",
"\n",
"| --model_max_length | --per_device_train_batch_size | GPU Memory |\n",
"|-----------------|------------|--------------------|\n",
"| 512 | 4 | 24g |\n",
"| 1024 | 3 | 24g |\n",
"| 512 | 8 | 32g |\n",
"| 1024 | 6 | 32g |\n",
"\n",
"You can use our recommended saving parameters, or you can save by epoch by just setting `--save_strategy \"epoch\"` if you prefer to save at each epoch stage. `--save_total_limit` means the limit on the number of saved checkpoints.\n",
"\n",
"For other parameters, such as `--weight_decay` and `--adam_beta2`, we recommend using the values we provided blow.\n",
"\n",
"Setting the parameters `--gradient_checkpointing` and `--lazy_preprocess` is to save GPU memory.\n",
"\n",
"The parameters for the trained Lora module will be saved in the **output_qwen** folder."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7ab0581e-be85-45e6-a5b7-af9c42ea697b",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"tags": []
},
"outputs": [],
"source": [
"!python ../../../finetune/finetune.py \\\n",
" --model_name_or_path \"Qwen/Qwen-7B-Chat/\"\\\n",
" --data_path \"medical_sft.json\"\\\n",
" --bf16 \\\n",
" --output_dir \"output_qwen\" \\\n",
" --num_train_epochs 4\\\n",
" --per_device_train_batch_size 4 \\\n",
" --per_device_eval_batch_size 3 \\\n",
" --gradient_accumulation_steps 16 \\\n",
" --evaluation_strategy \"no\" \\\n",
" --save_strategy \"epoch\" \\\n",
" --save_steps 3000 \\\n",
" --save_total_limit 10 \\\n",
" --learning_rate 1e-5 \\\n",
" --weight_decay 0.1 \\\n",
" --adam_beta2 0.95 \\\n",
" --warmup_ratio 0.01 \\\n",
" --lr_scheduler_type \"cosine\" \\\n",
" --logging_steps 10 \\\n",
" --model_max_length 512 \\\n",
" --gradient_checkpointing \\\n",
" --lazy_preprocess \\\n",
" --use_lora"
]
},
{
"cell_type": "markdown",
"id": "5e6f28aa-1772-48ce-aa15-8cf29e7d67b5",
"metadata": {},
"source": [
"## Merge Weights\n",
"\n",
"The LoRA training only saves the adapter parameters. You can load the fine-tuned model and merge weights as shown below:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "4fd5ef2a-34f9-4909-bebe-7b3b086fd16a",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-01-26T02:46:14.585746Z",
"iopub.status.busy": "2024-01-26T02:46:14.585089Z",
"iopub.status.idle": "2024-01-26T02:47:08.095464Z",
"shell.execute_reply": "2024-01-26T02:47:08.094715Z",
"shell.execute_reply.started": "2024-01-26T02:46:14.585720Z"
},
"tags": []
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to \"AutoModelForCausalLM.from_pretrained\".\n",
"Try importing flash-attention for faster inference...\n",
"Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm\n",
"Loading checkpoint shards: 100%|██████████| 8/8 [00:06<00:00, 1.14it/s]\n"
]
}
],
"source": [
"from transformers import AutoModelForCausalLM\n",
"from peft import PeftModel\n",
"import torch\n",
"\n",
"model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-7B-chat/\", torch_dtype=torch.float16, device_map=\"auto\", trust_remote_code=True)\n",
"model = PeftModel.from_pretrained(model, \"output_qwen/\")\n",
"merged_model = model.merge_and_unload()\n",
"merged_model.save_pretrained(\"output_qwen_merged\", max_shard_size=\"2048MB\", safe_serialization=True)"
]
},
{
"cell_type": "markdown",
"id": "2e3f5b9f-63a1-4599-8d9b-a8d8f764838f",
"metadata": {},
"source": [
"The tokenizer files are not saved in the new directory in this step. You can copy the tokenizer files or use the following code:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "10fa5ea3-dd55-4901-86af-c045d4c56533",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-01-26T02:47:08.097051Z",
"iopub.status.busy": "2024-01-26T02:47:08.096744Z",
"iopub.status.idle": "2024-01-26T02:47:08.591289Z",
"shell.execute_reply": "2024-01-26T02:47:08.590665Z",
"shell.execute_reply.started": "2024-01-26T02:47:08.097029Z"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"('output_qwen_merged/tokenizer_config.json',\n",
" 'output_qwen_merged/special_tokens_map.json',\n",
" 'output_qwen_merged/qwen.tiktoken',\n",
" 'output_qwen_merged/added_tokens.json')"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import AutoTokenizer\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(\n",
" \"Qwen/Qwen-7B-chat/\",\n",
" trust_remote_code=True\n",
")\n",
"\n",
"tokenizer.save_pretrained(\"output_qwen_merged\")"
]
},
{
"cell_type": "markdown",
"id": "804b84d8",
"metadata": {},
"source": [
"## Test the Model\n",
"\n",
"After merging the weights, we can test the model as follows:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "dbae310c",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-01-26T02:48:29.995040Z",
"iopub.status.busy": "2024-01-26T02:48:29.994448Z",
"iopub.status.idle": "2024-01-26T02:48:41.677104Z",
"shell.execute_reply": "2024-01-26T02:48:41.676591Z",
"shell.execute_reply.started": "2024-01-26T02:48:29.995019Z"
},
"tags": []
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm\n",
"Loading checkpoint shards: 100%|██████████| 8/8 [00:04<00:00, 1.71it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"VDAC1（电压依赖性钙通道）是一种位于细胞膜上的钙离子通道，负责将细胞内的钙离子释放到细胞外。它在神经信号传导、肌肉收缩和血管舒张中发挥着重要作用。\n",
"\n",
"VDAC1通常由4个亚基组成，每个亚基都有不同的功能。其中，一个亚基是内腔部分，它与钙离子的结合有关；另一个亚基是外腔部分，它与离子通道的打开和关闭有关；第三个亚基是一层跨膜蛋白，它负责调节通道的开放程度；最后一个亚基是一个膜骨架连接器，它帮助维持通道的结构稳定性。\n",
"\n",
"除了钙离子外，VDAC1还能够接收钾离子和氯离子等其他离子，并将其从细胞内释放到细胞外。此外，VDAC1还参与了许多细胞代谢反应，例如脂肪酸合成和糖原分解等。\n",
"\n",
"总的来说，VDAC1是细胞膜上的一种重要离子通道，其作用涉及到许多重要的生物学过程。\n"
]
}
],
"source": [
"from transformers import AutoModelForCausalLM, AutoTokenizer\n",
"from transformers.generation import GenerationConfig\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(\"output_qwen_merged\", trust_remote_code=True)\n",
"model = AutoModelForCausalLM.from_pretrained(\n",
" \"output_qwen_merged\",\n",
" device_map=\"auto\",\n",
" trust_remote_code=True\n",
").eval()\n",
"\n",
"response, history = model.chat(tokenizer, \"什么是VDAC1？\", history=None)\n",
"print(response)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "987f524d-6918-48ae-a730-f285cf6f8416",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

411
recipes/applications/retrieval/retrieval.ipynb Normal file
View File

@@ -0,0 +1,411 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "245ab07a-fb2f-4cf4-ab9a-5c05a9b44daa",
"metadata": {},
"source": [
"# LangChain retrieval knowledge base Q&A based on Qwen-7B-Chat"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e8df2cb7-a69c-4231-9596-4c871d893633",
"metadata": {},
"source": [
"This notebook introduces a question-answering application based on a local knowledge base using Qwen-7B-Chat with langchain. The goal is to establish a knowledge base Q&A solution that is friendly to many scenarios and open-source models, and that can run offline. The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the LLM (Large Language Model) to generate an answer."
]
},
{
"cell_type": "markdown",
"id": "92e9c81a-45c7-4c12-91af-3c5dd52f63bb",
"metadata": {},
"source": [
"## Preparation"
]
},
{
"cell_type": "markdown",
"id": "84cfcf88-3bef-4412-a658-4eaefeb6502a",
"metadata": {},
"source": [
"Download Qwen-7B-Chat\n",
"\n",
"Firstly, we need to download the model. You can use the snapshot_download that comes with modelscope to download the model to a specified directory."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9c1f9ded-8035-42c7-82c7-444ce06572bc",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!pip install modelscope"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7c26225c-c958-429e-b81d-2de9820670c2",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"tags": []
},
"outputs": [],
"source": [
"from modelscope.hub.snapshot_download import snapshot_download\n",
"snapshot_download(\"Qwen/Qwen-7B-Chat\",cache_dir='/tmp/models') "
]
},
{
"cell_type": "markdown",
"id": "e8f51796-49fa-467d-a825-ae9a281eb3fd",
"metadata": {},
"source": [
"Download the dependencies for langchain and Qwen."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "87fe1023-644f-4610-afaf-0b7cddc30d60",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"tags": []
},
"outputs": [],
"source": [
"!pip install langchain==0.0.187 dashscope==1.0.4 sentencepiece==0.1.99 cpm_kernels==1.0.11 nltk==3.8.1 sentence_transformers==2.2.2 unstructured==0.6.5 faiss-cpu==1.7.4 icetk==0.0.7"
]
},
{
"cell_type": "markdown",
"id": "853cdfa4-a2ce-4baa-919a-b9e2aecd2706",
"metadata": {},
"source": [
"Download the retrieval document."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8ba800dc-311d-4a83-8115-f05b09b39ffd",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/LLM_Survey_Chinese.pdf.txt"
]
},
{
"cell_type": "markdown",
"id": "07e923b3-b7ae-4983-abeb-2ce115566f15",
"metadata": {},
"source": [
"Download the text2vec model, for Chinese in our case."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9a07cd8d-3cec-40f6-8d2b-eb111aaf1164",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/qwen_recipes/GanymedeNil_text2vec-large-chinese.tar.gz\n",
"!tar -zxvf GanymedeNil_text2vec-large-chinese.tar.gz -C /tmp"
]
},
{
"cell_type": "markdown",
"id": "dc483af0-170e-4e61-8d25-a336d1592e34",
"metadata": {},
"source": [
"## Try out the model \n",
"\n",
"Load the Qwen-7B-Chat model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c112cf82-0447-46c4-9c32-18f243c0a686",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"tags": []
},
"outputs": [],
"source": [
"from abc import ABC\n",
"from langchain.llms.base import LLM\n",
"from typing import Any, List, Mapping, Optional\n",
"from langchain.callbacks.manager import CallbackManagerForLLMRun\n",
"from transformers import AutoModelForCausalLM, AutoTokenizer\n",
"\n",
"model_path=\"/tmp/models/Qwen/Qwen-7B-Chat\"\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)\n",
"model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True).half().cuda()\n",
"model.eval()\n",
"\n",
"class Qwen(LLM, ABC):\n",
" max_token: int = 10000\n",
" temperature: float = 0.01\n",
" top_p = 0.9\n",
" history_len: int = 3\n",
"\n",
" def __init__(self):\n",
" super().__init__()\n",
"\n",
" @property\n",
" def _llm_type(self) -> str:\n",
" return \"Qwen\"\n",
"\n",
" @property\n",
" def _history_len(self) -> int:\n",
" return self.history_len\n",
"\n",
" def set_history_len(self, history_len: int = 10) -> None:\n",
" self.history_len = history_len\n",
"\n",
" def _call(\n",
" self,\n",
" prompt: str,\n",
" stop: Optional[List[str]] = None,\n",
" run_manager: Optional[CallbackManagerForLLMRun] = None,\n",
" ) -> str:\n",
" response, _ = model.chat(tokenizer, prompt, history=[])\n",
" return response\n",
" \n",
" @property\n",
" def _identifying_params(self) -> Mapping[str, Any]:\n",
" \"\"\"Get the identifying parameters.\"\"\"\n",
" return {\"max_token\": self.max_token,\n",
" \"temperature\": self.temperature,\n",
" \"top_p\": self.top_p,\n",
" \"history_len\": self.history_len}\n",
" "
]
},
{
"cell_type": "markdown",
"id": "382ed433-870f-424e-b074-210ea6f84b70",
"metadata": {},
"source": [
"Specify the txt file that needs retrieval for knowledge-based Q&A."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "14be706b-4a7d-4906-9369-1f03c6c99854",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"tags": []
},
"outputs": [],
"source": [
"import os\n",
"import torch\n",
"import argparse\n",
"from langchain.vectorstores import FAISS\n",
"from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
"from typing import List, Tuple\n",
"import numpy as np\n",
"from langchain.document_loaders import TextLoader\n",
"from chinese_text_splitter import ChineseTextSplitter\n",
"from langchain.docstore.document import Document\n",
"from langchain.prompts.prompt import PromptTemplate\n",
"from langchain.chains import RetrievalQA\n",
"\n",
"\n",
"def load_file(filepath, sentence_size=100):\n",
" loader = TextLoader(filepath, autodetect_encoding=True)\n",
" textsplitter = ChineseTextSplitter(pdf=False, sentence_size=sentence_size)\n",
" docs = loader.load_and_split(textsplitter)\n",
" write_check_file(filepath, docs)\n",
" return docs\n",
"\n",
"\n",
"def write_check_file(filepath, docs):\n",
" folder_path = os.path.join(os.path.dirname(filepath), \"tmp_files\")\n",
" if not os.path.exists(folder_path):\n",
" os.makedirs(folder_path)\n",
" fp = os.path.join(folder_path, 'load_file.txt')\n",
" with open(fp, 'a+', encoding='utf-8') as fout:\n",
" fout.write(\"filepath=%s,len=%s\" % (filepath, len(docs)))\n",
" fout.write('\\n')\n",
" for i in docs:\n",
" fout.write(str(i))\n",
" fout.write('\\n')\n",
" fout.close()\n",
"\n",
" \n",
"def seperate_list(ls: List[int]) -> List[List[int]]:\n",
" lists = []\n",
" ls1 = [ls[0]]\n",
" for i in range(1, len(ls)):\n",
" if ls[i - 1] + 1 == ls[i]:\n",
" ls1.append(ls[i])\n",
" else:\n",
" lists.append(ls1)\n",
" ls1 = [ls[i]]\n",
" lists.append(ls1)\n",
" return lists\n",
"\n",
"\n",
"class FAISSWrapper(FAISS):\n",
" chunk_size = 250\n",
" chunk_conent = True\n",
" score_threshold = 0\n",
" \n",
" def similarity_search_with_score_by_vector(\n",
" self, embedding: List[float], k: int = 4\n",
" ) -> List[Tuple[Document, float]]:\n",
" scores, indices = self.index.search(np.array([embedding], dtype=np.float32), k)\n",
" docs = []\n",
" id_set = set()\n",
" store_len = len(self.index_to_docstore_id)\n",
" for j, i in enumerate(indices[0]):\n",
" if i == -1 or 0 < self.score_threshold < scores[0][j]:\n",
" # This happens when not enough docs are returned.\n",
" continue\n",
" _id = self.index_to_docstore_id[i]\n",
" doc = self.docstore.search(_id)\n",
" if not self.chunk_conent:\n",
" if not isinstance(doc, Document):\n",
" raise ValueError(f\"Could not find document for id {_id}, got {doc}\")\n",
" doc.metadata[\"score\"] = int(scores[0][j])\n",
" docs.append(doc)\n",
" continue\n",
" id_set.add(i)\n",
" docs_len = len(doc.page_content)\n",
" for k in range(1, max(i, store_len - i)):\n",
" break_flag = False\n",
" for l in [i + k, i - k]:\n",
" if 0 <= l < len(self.index_to_docstore_id):\n",
" _id0 = self.index_to_docstore_id[l]\n",
" doc0 = self.docstore.search(_id0)\n",
" if docs_len + len(doc0.page_content) > self.chunk_size:\n",
" break_flag = True\n",
" break\n",
" elif doc0.metadata[\"source\"] == doc.metadata[\"source\"]:\n",
" docs_len += len(doc0.page_content)\n",
" id_set.add(l)\n",
" if break_flag:\n",
" break\n",
" if not self.chunk_conent:\n",
" return docs\n",
" if len(id_set) == 0 and self.score_threshold > 0:\n",
" return []\n",
" id_list = sorted(list(id_set))\n",
" id_lists = seperate_list(id_list)\n",
" for id_seq in id_lists:\n",
" for id in id_seq:\n",
" if id == id_seq[0]:\n",
" _id = self.index_to_docstore_id[id]\n",
" doc = self.docstore.search(_id)\n",
" else:\n",
" _id0 = self.index_to_docstore_id[id]\n",
" doc0 = self.docstore.search(_id0)\n",
" doc.page_content += \" \" + doc0.page_content\n",
" if not isinstance(doc, Document):\n",
" raise ValueError(f\"Could not find document for id {_id}, got {doc}\")\n",
" doc_score = min([scores[0][id] for id in [indices[0].tolist().index(i) for i in id_seq if i in indices[0]]])\n",
" doc.metadata[\"score\"] = int(doc_score)\n",
" docs.append((doc, doc_score))\n",
" return docs\n",
"\n",
"\n",
"if __name__ == '__main__':\n",
" # load docs\n",
" filepath = 'LLM_Survey_Chinese.pdf.txt'\n",
" # LLM name\n",
" LLM_TYPE = 'qwen'\n",
" # Embedding model name\n",
" EMBEDDING_MODEL = 'text2vec'\n",
" # 基于上下文的prompt模版，请务必保留\"{question}\"和\"{context_str}\"\n",
" PROMPT_TEMPLATE = \"\"\"已知信息：\n",
" {context_str} \n",
" 根据上述已知信息，简洁和专业的来回答用户的问题。如果无法从中得到答案，请说 “根据已知信息无法回答该问题” 或 “没有提供足够的相关信息”，不允许在答案中添加编造成分，答案请使用中文。 问题是：{question}\"\"\"\n",
" # Embedding running device\n",
" EMBEDDING_DEVICE = \"cuda\"\n",
" # return top-k text chunk from vector store\n",
" VECTOR_SEARCH_TOP_K = 3\n",
" # 文本分句长度\n",
" SENTENCE_SIZE = 50\n",
" CHAIN_TYPE = 'stuff'\n",
" llm_model_dict = {\n",
" \"qwen\": QWen,\n",
" }\n",
" embedding_model_dict = {\n",
" \"text2vec\": \"/tmp/GanymedeNil_text2vec-large-chinese\",\n",
" }\n",
" print(\"loading model start\")\n",
" llm = llm_model_dict[LLM_TYPE]()\n",
" embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[EMBEDDING_MODEL],model_kwargs={'device': EMBEDDING_DEVICE})\n",
" print(\"loading model done\")\n",
"\n",
" print(\"loading documents start\")\n",
" docs = load_file(filepath, sentence_size=SENTENCE_SIZE)\n",
" print(\"loading documents done\")\n",
"\n",
" print(\"embedding start\")\n",
" docsearch = FAISSWrapper.from_documents(docs, embeddings)\n",
" print(\"embedding done\")\n",
"\n",
" print(\"loading qa start\")\n",
" prompt = PromptTemplate(\n",
" template=PROMPT_TEMPLATE, input_variables=[\"context_str\", \"question\"]\n",
" )\n",
"\n",
" chain_type_kwargs = {\"prompt\": prompt, \"document_variable_name\": \"context_str\"}\n",
" qa = RetrievalQA.from_chain_type(\n",
" llm=llm,\n",
" chain_type=CHAIN_TYPE, \n",
" retriever=docsearch.as_retriever(search_kwargs={\"k\": VECTOR_SEARCH_TOP_K}), \n",
" chain_type_kwargs=chain_type_kwargs)\n",
" print(\"loading qa done\")\n",
"\n",
" query = \"大模型指令微调有好的策略？\" \n",
" print(qa.run(query))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.15"
}
},
"nbformat": 4,
"nbformat_minor": 5
}