--- title: nlp_palm2.0_text-generation_meeting_title_chinese-base canonical_url: "https://www.modelscope.cn/models/damo/nlp_palm2.0_text-generation_meeting_title_chinese-base" md_url: "https://www.modelscope.cn/models/damo/nlp_palm2.0_text-generation_meeting_title_chinese-base.md" repository: damo/nlp_palm2.0_text-generation_meeting_title_chinese-base chinese_name: "PALM 2.0会议子标题生成-中文-base-ICASSP2023-MUG-Track3" last_updated: 2024-12-05 license: "Apache License 2.0" pipeline_tag: text-generation tasks: - text-generation model_type: - palm library_name: - pytorch frameworks: - pytorch language: - ch domain: - nlp downloads: 477 stars: 3 tags: - transformer - AliceMind - Alibaba - "volume:abs/2004.07159" --- # nlp_palm2.0_text-generation_meeting_title_chinese-base > nlp_palm2.0_text-generation_meeting_title_chinese-base - damo 在 ModelScope 开源的模型。达摩PALM 2.0中文会议子标题生成base模型，ICASSP2023-MUG-Track3 Topic Title Generation (TTG)的基线模型 damo/nlp_palm2.0_text-generation_meeting_title_chinese-base 是 ModelScope 魔搭社区上的text-generation模型，采用 Apache License 2.0 许可。 - **Repository**: damo/nlp_palm2.0_text-generation_meeting_title_chinese-base - **License**: Apache License 2.0 - **Tasks**: text-generation - **Tags**: transformer, AliceMind, Alibaba, volume:abs/2004.07159 - **Downloads**: 477 - **Stars**: 3 - **Last updated**: 2024-12-05 Source: https://www.modelscope.cn/models/damo/nlp_palm2.0_text-generation_meeting_title_chinese-base --- # ICASSP2023 MUG Challenge Track3 子标题生成Baseline ## 赛事及背景介绍 随着数字化经济的进一步发展，越来越多的企业开始将现代信息网络作为数据资源的主要载体，并通过网络通信技术进行数据传输。同时，疫情也促使越来越多行业逐步将互联网作为主要的信息交流和分享的方式。以往的研究表明，会议记录的口语语言处理（SLP）技术如关键词提取和摘要，对于信息的提取、组织和排序至关重要，可以显著提高用户对重要信息的掌握效率。 本项目源自于ICASSP2023信号处理大挑战的通用会议理解及生成挑战赛（MUG challenge），赛事构建并发布了目前为止规模最大的中文会议数据集，并基于会议人工转写结果进行了多项SLP任务的标注；目标是推动SLP在会议文本处理场景的研究并应对其中的多项关键挑战，包括 人人交互场景下多样化的口语现象、会议场景下的长篇章文档建模 等。 赛事报名页面： https://modelscope.cn/competition/12/summary - 话题分割 https://modelscope.cn/competition/13/summary - 抽取式摘要 https://modelscope.cn/competition/14/summary - 话题子标题生成 https://modelscope.cn/competition/17/summary - 行动项抽取 https://modelscope.cn/competition/18/summary - 关键词抽取 基线模型训练及推理： https://github.com/alibaba-damo-academy/SpokenNLP ## PALM中文生成模型 [PALM预训练语言生成模型]()是针对实际场景中常见的文本生成需求所设计的一个模型。模型利用大量无监督数据，通过结合自编码和自回归任务进行预训练，更贴合下游生成任务所同时需要的理解和生成能力。本模型是PALM通用预训练生成模型，在竞赛子标题生成数据集上进行finetune得到的会议子标题生成模型。PALM模型介绍，详见：[PALM:Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation](https://arxiv.org/abs/2004.07159) ## 期望模型使用方式以及适用范围 本模型主要用于给输入文档生成摘要内容。用户可以自行尝试各种输入文档。具体调用方式请参考代码示例。 ### 如何使用 在安装完成ModelScope-library之后即可使用text-generation的能力 #### 推理代码范例 ```python from modelscope.pipelines import pipeline from modelscope.utils.constant import Tasks from modelscope.outputs import OutputKeys input = '哎大家好啊欢迎大家准时来参加我们的会议，今天我们会议的主题呢是如何提升我们这个洗发水儿的品牌影响力啊。我们现在是主要的产品是这个霸王防脱洗发水儿，现在请大家发表自己的意见啊欢迎大家努力的发表，请先从这位男士开始。是这个。' text_summary = pipeline(Tasks.text_generation, model='damo/nlp_palm2.0_text-generation_meeting_title_chinese-base', external_engine_for_llm=False) result = text_summary(input) print('输入文本:\n' + input + '\n') print('文本生成结果:\n' + result[OutputKeys.TEXT]) ``` ### 模型局限性以及可能的偏差 模型在新闻相关数据集上训练，在新闻等类似文章上摘要生成性能较好，其他垂直领域效果可能会有所下降。 ## 训练数据介绍 本模型是由竞赛子标题数据集训练得到的， 具体数据可以参考右侧标签栏 ## 模型训练流程 用户可以基于这个PALM预训练底座模型进一步优化，训练代码如下，更详细代码可参考[alimeeting4mug](https://github.com/alibaba-damo-academy/SpokenNLP/blob/main/alimeeting4mug/run_palm_topic_title_generation.sh) ```python import tempfile import os from modelscope.msdatasets import MsDataset from modelscope.metainfo import Trainers from modelscope.trainers import build_trainer from modelscope.utils.hub import read_config from modelscope.metainfo import Metrics, Trainers from datasets import Dataset # ---------------------------- Train --------------------------------- # 用户自己数据集构造 tmp_dir = "./sub_title_generation_models/" data_pth = "./ttg_transformer" def load_local_data(data_pth): train_dataset_dict = {"src_txt": [], "tgt_txt": []} with open(os.path.join(data_pth, "train.source"), "r") as f: for line in f: train_dataset_dict["src_txt"].append(line.strip()) with open(os.path.join(data_pth, "train.target"), "r") as f: for line in f: train_dataset_dict["tgt_txt"].append(line.strip()) eval_dataset_dict = {"src_txt": [], "tgt_txt": []} with open(os.path.join(data_pth, "val.source"), "r") as f: for line in f: eval_dataset_dict["src_txt"].append(line.strip()) with open(os.path.join(data_pth, "val.target"), "r") as f: for line in f: eval_dataset_dict["tgt_txt"].append(line.strip()) return train_dataset_dict, eval_dataset_dict train_dataset_dict, eval_dataset_dict = load_local_data(data_pth) train_dataset = MsDataset(Dataset.from_dict(train_dataset_dict)) eval_dataset = MsDataset(Dataset.from_dict(eval_dataset_dict)) num_warmup_steps = 500 def noam_lambda(current_step: int): current_step += 1 return min(current_step**(-0.5), current_step * num_warmup_steps**(-1.5)) # 可以在代码修改 configuration 的配置 def cfg_modify_fn(cfg): cfg.train.lr_scheduler = { 'type': 'LambdaLR', 'lr_lambda': noam_lambda, 'options': { 'by_epoch': False } } cfg.preprocessor = { "type": "text-gen-tokenizer", "sequence_length": 512 } cfg.train.optimizer = { "type": "AdamW", "lr": 5e-4, "options": {} } cfg.train.hooks[-1] = { "type": "EvaluationHook", "by_epoch": True, "interval": 1 } cfg.train.max_epochs = 10 cfg.train.dataloader = { "batch_size_per_gpu": 16, "workers_per_gpu": 1 } return cfg kwargs = dict( model='damo/nlp_palm2.0_pretrained_chinese-base', train_dataset=train_dataset, eval_dataset=eval_dataset, work_dir=tmp_dir, cfg_modify_fn=cfg_modify_fn) trainer = build_trainer( name=Trainers.text_generation_trainer, default_args=kwargs) trainer.train() """ # ---------------------------- Evaluation --------------------------------- from modelscope.pipelines import pipeline from modelscope.utils.constant import Tasks from modelscope.outputs import OutputKeys from modelscope.metrics import TextGenerationMetric text_generation = pipeline(Tasks.text_generation, model=os.path.join(tmp_dir, "output"), sequence_length=512) m = TextGenerationMetric() outputs = {'preds': []} inputs = {'tgts': []} with open(os.path.join(data_pth, "val.source"), "r") as f: for line in f: output = text_generation(line.strip())['text'] outputs['preds'].append(output) with open(os.path.join(data_pth, "val.target"), "r") as f: for line in f: inputs['tgts'].append(line.strip()) m.add(outputs, inputs) print(f'{m.preds[:10]}\n{m.tgts[:10]}') print(m.evaluate()) # ---------------------------- Inference --------------------------------- output_list = [] with open(os.path.join(data_pth, "test.source"), "r") as f: for line in f: output = text_generation(line.strip())['text'] output_list.append(line.strip().replace("\t", " ") + "\t" + output) with open(os.path.join(tmp_dir, "test_predict_result.txt"), "w") as f: f.write("\n".join(output_list)) """ ``` ### 训练tips * 超参数调节主要是lr和epoch，可以在cfg_modify_fn里修改 * 生成长度短的数据集训练轮数可以小一些，在10～20epoch之间，生成长度长的数据集需要更多的轮数，如30～50epoch * 生成所需要的数据集量比较大，如果任务难度简单，则1w～10w即可，生成难度难的任务需要更多数据 ## 数据评估及结果 模型在竞赛dev集上评估结果 | Rouge-1 | Rouge-L | |----------|-------- | | 31.28 | 29.43 | ### 相关论文以及引用信息 如果我们的模型对您有帮助，请您引用我们的文章： ``` @inproceedings{bi-etal-2020-palm, title = "{PALM}: Pre-training an Autoencoding & Autoregressive Language Model for Context-conditioned Generation", author = "Bi, Bin and Li, Chenliang and Wu, Chen and Yan, Ming and Wang, Wei and Huang, Songfang and Huang, Fei and Si, Luo", booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)", month = nov, year = "2020", address = "Online", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2020.emnlp-main.700", doi = "10.18653/v1/2020.emnlp-main.700", pages = "8681--8691"} ```