该模块聚焦NLP模型可解释性评估，提供3个任务的评测数据与指标，含3种证据抽取方法，在多种模型结构上验证了模型复杂度和参数规模的影响。平台需Linux等环境及相关库，通过数据准备、模型训练、重要度分数获取等步骤运行，最终生成含模型准确率及可解释评测指标的评估报告。
深度学习模型在很多NLP任务上已经取得巨大成功，但其常被当作一个黑盒使用，内部预测机制对使用者是不透明的。这使得深度学习模型结果不被人信任，增加落地难度，尤其是在医疗、法律等特殊领域。同时，当模型出现效果不好或鲁棒性差等问题时，由于不了解其内部机制，导致很难对模型进行优化。近期，深度学习模型的可解释性被越来越多的人关注。但模型的可解释性评估还不够完善，本模块提供了3个NLP任务的评测数据和相关评测指标，旨在评估模型的可解释性。模块包含以下功能：
1. 完善可解释性评估体系，提供了评测数据和对应的评测指标 2. 提供了3种典型的证据抽取方法，分别是基于注意力（attention-based）、梯度（gradient-based）和线性模型（LIME）的证据抽取方法，并在LSTM、Transformer（RoBERTa-base和RoBERTa-large）等常用模型网络结构上完成实验验证，分别验证模型结构复杂度、模型参数规模对模型可解释的影响 3. 提供模型较全面的评估报告，含模型本身准确率等效果、以及在3个可解释评测指标上的结果
代码运行需要 Linux 主机，Python 3.8（推荐，其他低版本未测试过） 和 PaddlePaddle 2.1 以上版本。
除此之外，需要使用支持 GPU 的硬件环境。
需要安装GPU版的PaddlePaddle，以及一些默认的飞桨依赖。
更多关于 PaddlePaddle 的安装教程、使用方法等请参考官方文档.
!pip3 install paddlepaddle-gpu !pip3 install paddlenlp==2.2.4!pip3 install paddle-ernie
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Requirement already satisfied: paddlepaddle-gpu in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (2.2.2) Requirement already satisfied: Pillow in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu) (7.1.2) Requirement already satisfied: astor in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu) (0.8.1) Requirement already satisfied: requests>=2.20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu) (2.22.0) Requirement already satisfied: numpy>=1.13 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu) (1.16.4) Requirement already satisfied: decorator in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu) (4.4.2) Requirement already satisfied: protobuf>=3.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu) (3.14.0) Requirement already satisfied: six in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu) (1.16.0) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu) (3.0.4) Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu) (2.8) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu) (1.25.6) Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu) (2019.9.11) Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Requirement already satisfied: paddlenlp==2.2.4 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (2.2.4) Requirement already satisfied: colorama in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp==2.2.4) (0.4.4) Requirement already satisfied: jieba in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp==2.2.4) (0.42.1) Requirement already satisfied: seqeval in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp==2.2.4) (1.2.2) Requirement already satisfied: colorlog in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp==2.2.4) (4.1.0) Requirement already satisfied: h5py in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp==2.2.4) (2.9.0) Requirement already satisfied: multiprocess in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp==2.2.4) (0.70.11.1) Requirement already satisfied: six in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from h5py->paddlenlp==2.2.4) (1.16.0) Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from h5py->paddlenlp==2.2.4) (1.16.4) Requirement already satisfied: dill>=0.3.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from multiprocess->paddlenlp==2.2.4) (0.3.3) Requirement already satisfied: scikit-learn>=0.21.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from seqeval->paddlenlp==2.2.4) (0.22.1) Requirement already satisfied: joblib>=0.11 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp==2.2.4) (0.14.1) Requirement already satisfied: scipy>=0.17.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp==2.2.4) (1.3.0) Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Requirement already satisfied: paddle-ernie in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (0.2.0.dev1) Requirement already satisfied: requests in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddle-ernie) (2.22.0) Requirement already satisfied: pathlib2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddle-ernie) (2.3.7.post1) Requirement already satisfied: tqdm in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddle-ernie) (4.64.0) Requirement already satisfied: six in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pathlib2->paddle-ernie) (1.16.0) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->paddle-ernie) (1.25.6) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->paddle-ernie) (3.0.4) Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->paddle-ernie) (2019.9.11) Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->paddle-ernie) (2.8)
除 PaddlePaddle 及其依赖之外，还依赖其它第三方 Python 库，位于代码根目录的 requirements.txt 文件中。
可使用 pip 一键安装
%cd ./PaddleNLP-develop/examples/model_interpretation !pip3 install -r requirements.txt
/home/aistudio/PaddleNLP-develop/examples/model_interpretation Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Requirement already satisfied: nvgpu>=0.9.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (0.9.0) Requirement already satisfied: regex>=2021.11.10 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (2022.4.24) Requirement already satisfied: spacy>=2.3.7 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (3.2.4) Requirement already satisfied: tqdm>=4.62.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (4.64.0) Requirement already satisfied: visualdl>=2.2.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (2.2.3) Requirement already satisfied: ansi2html in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (1.7.0) Requirement already satisfied: psutil in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (5.7.2) Requirement already satisfied: flask-restful in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (0.3.9) Requirement already satisfied: requests in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (2.22.0) Requirement already satisfied: pandas in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (1.1.5) Requirement already satisfied: flask in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (1.1.1) Requirement already satisfied: tabulate in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (0.8.3) Requirement already satisfied: six in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (1.16.0) Requirement already satisfied: pynvml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (8.0.4) Requirement already satisfied: arrow in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (1.2.2) Requirement already satisfied: termcolor in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from nvgpu>=0.9.0->-r requirements.txt (line 1)) (1.1.0) Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (0.9.1) Requirement already satisfied: typer<0.5.0,>=0.3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (0.4.1) Requirement already satisfied: click<8.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (8.0.4) Requirement already satisfied: blis<0.8.0,>=0.4.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (0.7.7) Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (2.0.6) Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (1.0.2) Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (1.8.2) Requirement already satisfied: numpy>=1.15.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (1.16.4) Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (1.0.7) Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (2.4.3) Requirement already satisfied: setuptools in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (41.4.0) Requirement already satisfied: jinja2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (3.0.0) Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (3.0.9) Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (21.3) Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (3.3.0) Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (3.0.6) Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (8.0.15) Requirement already satisfied: pathy>=0.3.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (0.6.1) Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (3.10.0.2) Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from spacy>=2.3.7->-r requirements.txt (line 3)) (2.0.7) Requirement already satisfied: Flask-Babel>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.2.2->-r requirements.txt (line 5)) (1.0.0) Requirement already satisfied: Pillow>=7.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.2.2->-r requirements.txt (line 5)) (7.1.2) Requirement already satisfied: bce-python-sdk in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.2.2->-r requirements.txt (line 5)) (0.8.53) Requirement already satisfied: protobuf>=3.11.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.2.2->-r requirements.txt (line 5)) (3.14.0) Requirement already satisfied: shellcheck-py in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.2.2->-r requirements.txt (line 5)) (0.7.1.1) Requirement already satisfied: flake8>=3.7.9 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.2.2->-r requirements.txt (line 5)) (4.0.1) Requirement already satisfied: matplotlib in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.2.2->-r requirements.txt (line 5)) (2.2.3) Requirement already satisfied: pre-commit in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.2.2->-r requirements.txt (line 5)) (1.21.0) Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from catalogue<2.1.0,>=2.0.6->spacy>=2.3.7->-r requirements.txt (line 3)) (3.8.0) Requirement already satisfied: importlib-metadata in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from click<8.1.0->spacy>=2.3.7->-r requirements.txt (line 3)) (4.2.0) Requirement already satisfied: pycodestyle<2.9.0,>=2.8.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl>=2.2.2->-r requirements.txt (line 5)) (2.8.0) Requirement already satisfied: pyflakes<2.5.0,>=2.4.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl>=2.2.2->-r requirements.txt (line 5)) (2.4.0) Requirement already satisfied: mccabe<0.7.0,>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl>=2.2.2->-r requirements.txt (line 5)) (0.6.1) Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask->nvgpu>=0.9.0->-r requirements.txt (line 1)) (0.16.0) Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask->nvgpu>=0.9.0->-r requirements.txt (line 1)) (1.1.0) Requirement already satisfied: Babel>=2.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl>=2.2.2->-r requirements.txt (line 5)) (2.8.0) Requirement already satisfied: pytz in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl>=2.2.2->-r requirements.txt (line 5)) (2019.3) Requirement already satisfied: MarkupSafe>=2.0.0rc2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from jinja2->spacy>=2.3.7->-r requirements.txt (line 3)) (2.0.1) Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from packaging>=20.0->spacy>=2.3.7->-r requirements.txt (line 3)) (3.0.8) Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pathy>=0.3.5->spacy>=2.3.7->-r requirements.txt (line 3)) (5.2.1) Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->nvgpu>=0.9.0->-r requirements.txt (line 1)) (2019.9.11) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->nvgpu>=0.9.0->-r requirements.txt (line 1)) (1.25.6) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->nvgpu>=0.9.0->-r requirements.txt (line 1)) (3.0.4) Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->nvgpu>=0.9.0->-r requirements.txt (line 1)) (2.8) Requirement already satisfied: python-dateutil>=2.7.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from arrow->nvgpu>=0.9.0->-r requirements.txt (line 1)) (2.8.2) Requirement already satisfied: future>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from bce-python-sdk->visualdl>=2.2.2->-r requirements.txt (line 5)) (0.18.0) Requirement already satisfied: pycryptodome>=3.8.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from bce-python-sdk->visualdl>=2.2.2->-r requirements.txt (line 5)) (3.9.9) Requirement already satisfied: aniso8601>=0.82 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask-restful->nvgpu>=0.9.0->-r requirements.txt (line 1)) (9.0.1) Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->visualdl>=2.2.2->-r requirements.txt (line 5)) (1.1.0) Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->visualdl>=2.2.2->-r requirements.txt (line 5)) (0.10.0) Requirement already satisfied: aspy.yaml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.2.2->-r requirements.txt (line 5)) (1.3.0) Requirement already satisfied: cfgv>=2.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.2.2->-r requirements.txt (line 5)) (2.0.1) Requirement already satisfied: pyyaml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.2.2->-r requirements.txt (line 5)) (5.1.2) Requirement already satisfied: nodeenv>=0.11.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.2.2->-r requirements.txt (line 5)) (1.3.4) Requirement already satisfied: toml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.2.2->-r requirements.txt (line 5)) (0.10.0) Requirement already satisfied: virtualenv>=15.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.2.2->-r requirements.txt (line 5)) (16.7.9) Requirement already satisfied: identify>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.2.2->-r requirements.txt (line 5)) (1.4.10)
中文推荐使用ChnSentiCorp，英文推荐使用SST-2。本模块提供的中英文情感分析模型就是基于这两个数据集的。若修改训练数据集，请修改/model_interpretation/task/senti/pretrained_models/train.py (RoBERTa) 以及 /model_interpretation/task/senti/rnn/train.py (LSTM)。
使用paddlenlp框架自动缓存模型文件。
请运行download.sh自动下载
!chmod +x ./download.sh !./download.sh
评测数据样例位于/model_interpretation/data/目录下，每一行为一条JSON格式的数据。
id: 数据的编号，作为该条数据识别key； context：原文本数据； sent_token：原文本数据的标准分词，注意：golden证据是基于该分词的，预测证据也需要与该分词对应； sample_type： 数据的类性，分为原始数据（ori）和扰动数据（disturb）； rel_ids：与原始数据关联的扰动数据的id列表（只有原始数据有）；
首先，进入task/senti/pretrained_models目录
cd ./task/senti/pretrained_models
/home/aistudio/PaddleNLP-develop/examples/model_interpretation/task/senti/pretrained_models
修改run_train.sh中必要参数，然后运行run_train.sh
!chmod +x ./run_train.sh !./run_train.sh
等待训练完成即可。训练过程可查看pretrained_models/logs文件夹中的日志。
下一步就是给每一个token获取一个重要度分数。首先进入task/senti/目录
cd ..
/home/aistudio/PaddleNLP-develop/examples/model_interpretation/task/senti
修改run_inter.sh中必要参数，并运行。
!chmod +x ./run_inter.sh !./run_inter.sh
++ python3 ./saliency_map/sentiment_interpretable.py --language ch --base_model roberta_base --data_dir ../../data/senti_ch --vocab_path test --from_pretrained roberta-wwm-ext --batch_size 1 --init_checkpoint pretrained_models/saved_model_ch/roberta_base/model_900/model_state.pdparams --inter_mode attention --output_dir ./output/senti_ch.roberta_base --n-samples 200 --start_id 0 --eval /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py:312: UserWarning: You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default. "You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default." /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlenlp/transformers/funnel/modeling.py:30: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import Iterable[2022-04-28 14:25:35,448] [ INFO] - Downloading https://bj.bcebos.com/paddlenlp/models/transformers/roberta_base/vocab.txt and saved to /home/aistudio/.paddlenlp/models/roberta-wwm-ext[2022-04-28 14:25:35,449] [ INFO] - Downloading vocab.txt from https://bj.bcebos.com/paddlenlp/models/transformers/roberta_base/vocab.txt 100%|████████████████████████████████████████| 107k/107k [00:00<00:00, 2.07MB/s][2022-04-28 14:25:35,607] [ INFO] - Downloading https://paddlenlp.bj.bcebos.com/models/transformers/roberta_base/roberta_chn_base.pdparams and saved to /home/aistudio/.paddlenlp/models/roberta-wwm-ext[2022-04-28 14:25:35,607] [ INFO] - Downloading roberta_chn_base.pdparams from https://paddlenlp.bj.bcebos.com/models/transformers/roberta_base/roberta_chn_base.pdparams 100%|████████████████████████████████████████| 390M/390M [00:35<00:00, 11.4MB/s] 100it [00:17, 5.82it/s]
得到重要度分数后，我们就可以根据每个token的分数做证据抽取了。 首先我们去到证据抽取模块rationale_extraction目录。
cd ../../rationale_extraction/
/home/aistudio/PaddleNLP-develop/examples/model_interpretation/rationale_extraction
修改generate.sh和run_2_pred_senti_per.sh中必要参数，并运行。
!chmod +x ./generate.sh !./generate.sh
roberta_base_attention_ch num: 100 +++ python3 ./sentiment_pred.py --base_model roberta_base --data_dir ./rationale/senti/roberta_base_attention_ch/rationale_text/dev --output_dir ./prediction/senti/roberta_base_attention_ch/rationale_text/dev --vocab_path test --from_pretrained roberta-wwm-ext --batch_size 1 --init_checkpoint ../task/senti/pretrained_models/saved_model_ch/roberta_base/model_900/model_state.pdparams --inter_mode attention --n-samples 200 --language ch /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py:312: UserWarning: You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default. "You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default." /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlenlp/transformers/funnel/modeling.py:30: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import Iterable[2022-04-28 13:01:39,498] [ INFO] - Already cached /home/aistudio/.paddlenlp/models/roberta-wwm-ext/vocab.txt[2022-04-28 13:01:39,516] [ INFO] - Already cached /home/aistudio/.paddlenlp/models/roberta-wwm-ext/roberta_chn_base.pdparams load data 100 load model from ../task/senti/pretrained_models/saved_model_ch/roberta_base/model_900/model_state.pdparams 100it [00:12, 8.05it/s] +++ for RATIONAL_TYPE in '"rationale_text"' '"rationale_exclusive_text"' +++ [[ ch == \e\n ]] +++ [[ ch == \c\h ]] +++ [[ roberta_base == \r\o\b\e\r\t\a\_\b\a\s\e ]] +++ FROM_PRETRAIN=roberta-wwm-ext +++ CKPT=../task/senti/pretrained_models/saved_model_ch/roberta_base/model_900/model_state.pdparams +++ OUTPUT=./prediction/senti/roberta_base_attention_ch/rationale_exclusive_text/dev +++ '[' -d ./prediction/senti/roberta_base_attention_ch/rationale_exclusive_text/dev ']' +++ set -x +++ python3 ./sentiment_pred.py --base_model roberta_base --data_dir ./rationale/senti/roberta_base_attention_ch/rationale_exclusive_text/dev --output_dir ./prediction/senti/roberta_base_attention_ch/rationale_exclusive_text/dev --vocab_path test --from_pretrained roberta-wwm-ext --batch_size 1 --init_checkpoint ../task/senti/pretrained_models/saved_model_ch/roberta_base/model_900/model_state.pdparams --inter_mode attention --n-samples 200 --language ch /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py:312: UserWarning: You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default. "You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default." /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlenlp/transformers/funnel/modeling.py:30: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import Iterable[2022-04-28 13:02:11,937] [ INFO] - Already cached /home/aistudio/.paddlenlp/models/roberta-wwm-ext/vocab.txt[2022-04-28 13:02:11,954] [ INFO] - Already cached /home/aistudio/.paddlenlp/models/roberta-wwm-ext/roberta_chn_base.pdparams load data 100 load model from ../task/senti/pretrained_models/saved_model_ch/roberta_base/model_900/model_state.pdparams 100it [00:12, 7.70it/s] roberta_base_attention_ch_finished
最后我们使用evaluation目录下的评估模块，评估一下结果
cd ../evaluation/
/home/aistudio/PaddleNLP-develop/examples/model_interpretation/evaluation
以合理性评估为例，进入plausibility目录，修改run_f1.sh中必要的参数，运行run_f1.sh
cd ./plausibility/
/home/aistudio/PaddleNLP-develop/examples/model_interpretation/evaluation/plausibility
!chmod +x ./run_f1.sh !./run_f1.sh
roberta_base_attention_ch num 4.00 macor_f1: 46.2
中文情感分析评估报告样例：
| 模型 + 证据抽取方法 | 情感分析 | |||
| Acc | Macro-F1 | MAP | New_P | |
| LSTM + IG | 56.8 | 36.8 | 59.8 | 91.4 |
| RoBERTa-base + IG | 62.4 | 36.4 | 48.7 | 48.9 |
| RoBERTa-large + IG | 65.3 | 38.3 | 41.9 | 37.8 |
以上就是『2022语言与智能技术竞赛』- 情感可解释性评测的详细内容，更多请关注php中文网其它相关文章！
每个人都需要一台速度更快、更稳定的 PC。随着时间的推移，垃圾文件、旧注册表数据和不必要的后台进程会占用资源并降低性能。幸运的是，许多工具可以让 Windows 保持平稳运行。
广告![ThinkPHP5快速开发企业站点[全程实录]](https://img.php.cn/upload/course/000/000/068/6253d918a3ce7278.png)
收藏
收藏
Copyright 2014-2025 https://www.php.cn/ All Rights Reserved | php.cn | 湘ICP备2023035733号
651
