tutorials/How_to_run_regression_testing_for_LLMs.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D8ddKzAHSwTX"
      },
      "outputs": [],
      "source": [
        "!pip install evidently[llm]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jsqbwE0krKtd"
      },
      "outputs": [],
      "source": [
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HlkBSDJiUFWd"
      },
      "outputs": [],
      "source": [
        "from evidently import ColumnMapping\n",
        "from evidently.test_suite import TestSuite\n",
        "from evidently.descriptors import *\n",
        "from evidently.tests import *"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_6UYbEHttL0Q"
      },
      "source": [
        "# Semantic similarity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "my51mYzk_BFO"
      },
      "source": [
        "## Mock data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1IOTpIEoyGXn"
      },
      "source": [
        "Create a mock dataset with typical questions and reference answers:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hvASt8Z7wXYG"
      },
      "outputs": [],
      "source": [
        "data = [\n",
        "    [\"Hi there, how do I reset my password?\", \"To reset your password, click on 'Forgot Password' on the login page and follow the instructions sent to your registered email.\"],\n",
        "    [\"Where can I find my transaction history?\", \"You can view your transaction history by logging into your account and navigating to the 'Transaction History' section.\"],\n",
        "    [\"How do I add another user to my account?\", \"I am afraid it is not currently possible to add multiple users to the account.\"],\n",
        "    [\"Is it possible to link multiple bank accounts?\", \"Yes, you can link multiple bank accounts by going to 'Account Settings' and selecting 'Add Bank Account'.\"],\n",
        "    [\"Can I use your service for cryptocurrency transactions?\", \"Currently, our service does not support cryptocurrency transactions. Please check our website for updates regarding this feature.\"],\n",
        "    [\"Hi, can I get a detailed report of my monthly transactions?\", \"Yes, you can generate a detailed monthly report of your transactions by logging into your account, going to 'Transaction History', and selecting 'Generate Report'.\"],\n",
        "    [\"I am traveling to the US. Can I use the app there?\", \"Yes, you can use the app in the US just like you do at home. Ensure you have an internet connection.\"],\n",
        "    [\"How do I link my payment account to a new mobile number?\", \"To link a new mobile number, log in to your account, go to 'Account Settings', select 'Mobile Number', and follow the instructions to verify your new number.\"],\n",
        "    [\"Can I receive notifications for transactions in real-time?\", \"Yes, you can enable real-time notifications for transactions by going to 'Account Settings', then 'Notifications', and turning on 'Transaction Alerts'.\"],\n",
        "    [\"Hey, can I set up automatic transfers to my savings account?\", \"Yes, you can set up automatic transfers by going to 'Account Settings', selecting 'Automatic Transfers', and specifying the amount and frequency.\"],\n",
        "]\n",
        "\n",
        "columns = [\"question\", \"target_response\"]\n",
        "\n",
        "typical_questions_golden = pd.DataFrame(data, columns=columns)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0bCym1cEEglY"
      },
      "outputs": [],
      "source": [
        "pd.set_option('display.max_colwidth', None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bF5LUHh2Wefw"
      },
      "source": [
        "Preview the dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TuznKNSrw2E4"
      },
      "outputs": [],
      "source": [
        "typical_questions_golden.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1rtpFiMUyKpu"
      },
      "source": [
        "Let's imagine we generated a new set of answers (e.g., with a different prompt) to the same questions. We add a column \"new response\" to the dataset. To generate the complete dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MyKyAyivwap-"
      },
      "outputs": [],
      "source": [
        "data = [\n",
        "    [\"Hi there, how do I reset my password?\", \"To reset your password, click on 'Forgot Password' on the login page and follow the instructions sent to your registered email.\", \"To change your password, select 'Forgot Password' on the login screen and follow the steps sent to your registered email address.\"],\n",
        "    [\"Where can I find my transaction history?\", \"You can view your transaction history by logging into your account and navigating to the 'Transaction History' section.\", \"Log into your account and go to 'Transaction History' to see all your past transactions.\"],\n",
        "    [\"How do I add another user to my account?\", \"I am afraid it is not currently possible to add multiple users to the account.\", \"To add a secondary user, go to 'Account Settings', select 'Manage Users', and enter the details of the person you want to add. You can set permissions for their access.\"],\n",
        "    [\"Is it possible to link multiple bank accounts?\", \"Yes, you can link multiple bank accounts by going to 'Account Settings' and selecting 'Add Bank Account'.\", \"You can add multiple bank accounts by visiting 'Account Settings' and choosing 'Add Bank Account'.\"],\n",
        "    [\"Can I use your service for cryptocurrency transactions?\", \"Currently, our service does not support cryptocurrency transactions. Please check our website for updates regarding this feature.\", \"At the moment, cryptocurrency transactions are not supported by our service. Keep an eye on our website for future updates.\"],\n",
        "    [\"Hi, can I get a detailed report of my monthly transactions?\", \"Yes, you can generate a detailed monthly report of your transactions by logging into your account, going to 'Transaction History', and selecting 'Generate Report'.\", \"You can get a detailed monthly report by logging into your account, navigating to 'Transaction History', and clicking on 'Generate Report'.\"],\n",
        "    [\"I am traveling to the US. Can I use the app there?\", \"Yes, you can use the app in the US just like you do at home. Ensure you have an internet connection.\", \"The app will work in the US without any issues. Just make sure you have access to the internet.\"],\n",
        "    [\"How do I link my payment account to a new mobile number?\", \"To link a new mobile number, log in to your account, go to 'Account Settings', select 'Mobile Number', and follow the instructions to verify your new number.\", \"To add a new number, navigate to the 'Account Settings' section, select 'Mobile Number' and proceed with the steps to add and confirm the new number\"],\n",
        "    [\"Can I receive notifications for transactions in real-time?\", \"Yes, you can enable real-time notifications for transactions by going to 'Account Settings', then 'Notifications', and turning on 'Transaction Alerts'.\", \"To see your transacation, log into your account and navigate to the 'Transaction History' section.\"],\n",
        "    [\"Hey, can I set up automatic transfers to my savings account?\", \"Yes, you can set up automatic transfers by going to 'Account Settings', selecting 'Automatic Transfers', and specifying the amount and frequency.\", \"You can arrange automatic transfers by going to 'Account Settings', choosing 'Automatic Transfers', and setting the desired amount and frequency.\"],\n",
        "]\n",
        "\n",
        "columns = [\"question\", \"target_response\", \"new_response\"]\n",
        "\n",
        "typical_questions = pd.DataFrame(data, columns=columns)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cLvWkGWtWriF"
      },
      "source": [
        "Preview the dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pFoiCLZnw-DW"
      },
      "outputs": [],
      "source": [
        "typical_questions.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vDFTXNQH_Dst"
      },
      "source": [
        "## Example 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FBOQXPVfyScR"
      },
      "source": [
        "Let's compare new generations for typical questions against the golden set of answers using Semantic Similarity. We expect all answers to be similar (with a similarity of 0.9 or higher)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sq8evokczQwB"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestColumnValueMin(\n",
        "        column_name=SemanticSimilarity(\n",
        "        display_name=\"Response Similarity\").\n",
        "        on([\"target_response\", \"new_response\"]),\n",
        "        gte=0.9),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None,\n",
        "               current_data=typical_questions)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VR6BdSoCW20R"
      },
      "source": [
        "Explore the dataset with the added Semantic Similarity score:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ttpE2LiY9UGl"
      },
      "outputs": [],
      "source": [
        "test_suite.datasets()[1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m1ANsN5tW5Uw"
      },
      "source": [
        "Select only the examples where Semantic Similarity is < 0.9:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DSYbyjO6X9MV"
      },
      "outputs": [],
      "source": [
        "df = pd.DataFrame(test_suite.datasets()[1])\n",
        "\n",
        "filtered_df = df[df['Response Similarity'] < 0.9]\n",
        "\n",
        "filtered_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KBf8WbKoCM6o"
      },
      "source": [
        "## Example 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XQT448SjWH9k"
      },
      "source": [
        "Let's consider another set of test cases: these are example questions where our chatbot should escalate to a human support agent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RDC_YkbwCOYo"
      },
      "outputs": [],
      "source": [
        "data = [\n",
        "    [\"There is a transaction in my history that I did not make. What should I do?\", \"Hi there! Transferring you to a support agent to help.\", \"Hi there! Transferring you to a support agent to help.\"],\n",
        "    [\"My account was charged twice for the same transaction. How do I get a refund?\", \"Hello! I'll transfer you to a support agent who can assist with this issue.\", \"Hello! I'll transfer you to a support agent who can assist with this issue.\"],\n",
        "    [\"I think my account has been hacked. What should I do?\", \"Hi there! I'll connect you to a support agent immediately to secure your account.\", \"Hi there! I'll connect you to a support agent immediately to secure your account.\"],\n",
        "    [\"I have an ongoing subscription, why was my recent payment to a merchant declined?\", \"Hello! Let me transfer you to a support agent who can look into this for you.\", \"Your payment might have been declined due to insufficient funds, an expired card, or incorrect billing information. Please check your payment details and try again.\"],\n",
        "    [\"I received a notification of a large withdrawal that I didn't authorize. Can you help?\", \"Hi! I'll transfer you to a support agent who can help with this urgent matter.\", \"Hi! I'll transfer you to a support agent who can help with this urgent matter.\"],\n",
        "    [\"Why was my account suddenly locked?\", \"Hello! Let me connect you to a support agent to find out more and help you unlock your account.\", \"Hello! Let me connect you to a support agent to find out more and help you unlock your account.\"],\n",
        "    [\"I am having issues verifying my identity. What should I do?\", \"Hi! I'll transfer you to a support agent who can assist you with the verification process.\", \"Hi! I'll transfer you to a support agent who can assist you with the verification process.\"],\n",
        "    [\"My recent transfer is still pending after several days. Can you check it for me?\", \"Hi there! I'll connect you to a support agent who can investigate this issue.\", \"Hi there! I'll connect you to a support agent who can investigate this issue.\"],\n",
        "    [\"I received an email about suspicious activity on my account. Is it legitimate?\", \"Hello! Let me transfer you to a support agent to verify the email and secure your account.\", \"Hello! Let me transfer you to a support agent to verify the email and secure your account.\"],\n",
        "    [\"I need to dispute a charge on my statement. How do I proceed?\", \"Hi there! I'll connect you to a support agent who can help you with the dispute process.\", \"Hi there! I'll connect you to a support agent who can help you with the dispute process.\"],\n",
        "]\n",
        "\n",
        "columns = [\"question\", \"target_response\", \"new_response\"]\n",
        "\n",
        "agent_questions = pd.DataFrame(data, columns=columns)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rNf0odxEEqpJ"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestColumnValueMin(\n",
        "        column_name=SemanticSimilarity(\n",
        "        display_name=\"Response Similarity\").\n",
        "        on([\"target_response\", \"new_response\"]),\n",
        "        gte=0.9),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None,\n",
        "               current_data=agent_questions)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "poLVDuJQc8az"
      },
      "source": [
        "Explore the dataset with the added Semantic Similarity score:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6h256BbrEtP7"
      },
      "outputs": [],
      "source": [
        "test_suite.datasets()[1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rIHNneKTdAuH"
      },
      "source": [
        "Select only the examples where Semantic Similarity is < 0.9:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1aYMOj_4YIM2"
      },
      "outputs": [],
      "source": [
        "df = pd.DataFrame(test_suite.datasets()[1])\n",
        "\n",
        "filtered_df = df[df['Response Similarity'] < 0.9]\n",
        "\n",
        "filtered_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A346l54rBp9O"
      },
      "source": [
        "# Test Classification Quality"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpzDLdjqb-nB"
      },
      "source": [
        "This is often a classification problem: you classify incoming requests by scenario to decide which to answer automatically. Let's test it like a classification problem against a labeled dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i_RO3qxccLwB"
      },
      "source": [
        "A new mock dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MVHZ3RMlYbtz"
      },
      "outputs": [],
      "source": [
        "data = [\n",
        "    [\"How do I reset my password?\", \"automated\", \"automated\"],\n",
        "    [\"There is a transaction in my history that I did not make. What should I do?\", \"agent\", \"agent\"],\n",
        "    [\"Can I change the email address associated with my account?\", \"automated\", \"automated\"],\n",
        "    [\"My account was charged twice for the same transaction. How do I get a refund?\", \"agent\", \"agent\"],\n",
        "    [\"What are the fees for international transactions?\", \"automated\", \"automated\"],\n",
        "    [\"I think my account has been hacked. What should I do?\", \"automated\", \"automated\"],\n",
        "    [\"How can I update my billing information?\", \"automated\", \"automated\"],\n",
        "    [\"Why was my recent payment to a merchant declined?\", \"agent\", \"automated\"], # Error\n",
        "    [\"Can I receive notifications for transactions in real-time?\", \"automated\", \"automated\"],\n",
        "    [\"I received a notification of a large withdrawal that I didn't authorize. Can you help?\", \"agent\", \"agent\"],\n",
        "    [\"How do I contact customer support?\", \"automated\", \"automated\"],\n",
        "    [\"I have a strange transacation\", \"agent\", \"automated\"], # Error\n",
        "    [\"Why was my account suddenly locked?\", \"agent\", \"agent\"],\n",
        "    [\"How can I verify my account?\", \"automated\", \"automated\"],\n",
        "    [\"I am having issues verifying my identity. What should I do?\", \"agent\", \"agent\"],\n",
        "    [\"My recent transfer is still pending after several days. Can you check it for me?\", \"agent\", \"agent\"],\n",
        "    [\"I received an email about suspicious activity on my account. Is it legitimate?\", \"agent\", \"agent\"],\n",
        "    [\"How do I change my payment method?\", \"automated\", \"automated\"],\n",
        "    [\"I need to dispute a charge on my statement. How do I proceed?\", \"agent\", \"agent\"],\n",
        "    [\"What is 2-factor authentication and how do I enable it?\", \"automated\", \"automated\"],\n",
        "    [\"Where can I seee my transactions?\", \"automated\", \"automated\"],\n",
        "    [\"Can I change my phone number?\", \"automated\", \"automated\"],\n",
        "    [\"I think someone is using my card\", \"agent\", \"agent\"],\n",
        "    [\"Can I link multiple bank accounts?\", \"automated\", \"automated\"],\n",
        "    [\"Can I add another bank account?\", \"automated\", \"automated\"],\n",
        "    [\"Can I use your service for cryptocurrency transactions?\", \"automated\", \"automated\"],\n",
        "    [\"How do I apply for a credit card?\", \"automated\", \"agent\"],  # Error\n",
        "    [\"What should I do if I forgot my username?\", \"automated\", \"automated\"],\n",
        "    [\"Can I add a new phone number?\", \"automated\", \"automated\"],\n",
        "    [\"What is my tariff plan?\", \"automated\", \"automated\"],\n",
        "    [\"Can I pay with your card online?\", \"automated\", \"automated\"],\n",
        "    [\"Where can I see the payment fees?\", \"automated\", \"automated\"],\n",
        "    [\"Can I switch my tarrif?\", \"automated\", \"automated\"],\n",
        "    [\"Which countries do you support?\", \"automated\", \"automated\"],\n",
        "    [\"I want to set up spending limit on my card\", \"automated\", \"automated\"],\n",
        "    [\"What measures do you have in place for data security?\", \"automated\", \"automated\"],\n",
        "    [\"Can I set spending limits?\", \"automated\", \"automated\"],\n",
        "    [\"How do I add a secondary user to my account?\", \"automated\", \"automated\"],\n",
        "    [\"Is it possible to get a detailed report of my monthly transactions?\", \"automated\", \"automated\"],\n",
        "    [\"My account get locked, help\", \"agent\", \"agent\"],\n",
        "]\n",
        "\n",
        "columns = [\"question\", \"target\", \"prediction\"]\n",
        "\n",
        "classification_data = pd.DataFrame(data, columns=columns)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kWF7X0kwccJV"
      },
      "source": [
        "Preview:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F1kD-sS3wJ6A"
      },
      "outputs": [],
      "source": [
        "classification_data.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9M4s6V-WcOhq"
      },
      "source": [
        "Test accuracy:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cWdfYQOAYt9Q"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestAccuracyScore(gte=0.9),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None,\n",
        "               current_data=classification_data,\n",
        "               column_mapping=ColumnMapping(pos_label='agent'))\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZ4YuYHzcRVN"
      },
      "source": [
        "Test recall for \"agent\" class:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XbP2mgmHZIav"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestRecallScore(gte=0.9),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None,\n",
        "               current_data=classification_data,\n",
        "               column_mapping=ColumnMapping(pos_label='agent'))\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xPILSGcIMoR4"
      },
      "source": [
        "# Test Text Length  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SJoBzzs7cgSd"
      },
      "source": [
        "Test that all new responses are between 50 and 180 symbols in length:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "awjvkV80Ijzv"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestValueRange(\n",
        "        column_name=TextLength(\n",
        "        display_name=\"Response Length\").\n",
        "        on(\"new_response\"),\n",
        "        left=50, right=180),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None,\n",
        "               current_data=typical_questions)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xgoFQRnGcoPl"
      },
      "source": [
        "Test that all new responses are 150 symbols or shorter."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wtDhzll1Orut"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestColumnValueMax(\n",
        "        column_name=TextLength(\n",
        "        display_name=\"Response Length\").\n",
        "        on(\"new_response\"),\n",
        "        lte=150),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None,\n",
        "               current_data=typical_questions)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "y_tprY4smVnN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fhs2_U15ctfK"
      },
      "source": [
        "View the dataset with the length column added:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cor-fkVVxJSz"
      },
      "outputs": [],
      "source": [
        "test_suite.datasets()[1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YXxBGWvnczur"
      },
      "source": [
        "Select only the examples where the Text Length is > 150:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "orlN9EcUc3sF"
      },
      "outputs": [],
      "source": [
        "df = pd.DataFrame(test_suite.datasets()[1])\n",
        "\n",
        "filtered_df = df[df['Response Length'] > 150]\n",
        "\n",
        "filtered_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C-Clh4U4RMLI"
      },
      "source": [
        "# Test Competitor Mentions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LWgnvKO6dYCE"
      },
      "source": [
        "Let's consider another set of test cases: these are example questions that mention competitors. Our chatbot's response should mention our own products."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YQU1nAKWROzr"
      },
      "outputs": [],
      "source": [
        "data = [\n",
        "    [\"Why should I switch from YetAnotherFinCo to OurFinCo?\", \"I can only consult about OurFinCo products. OurFinCo offers competitive fees, robust security measures, and excellent customer support.\", \"I can only consult about OurFinCo products. OurFinCo offers competitive fees, robust security measures, and excellent customer support.\"],\n",
        "    [\"How do your fees compare to AnotherFinCo?\", \"I can only provide information about OurFinCo products. OurFinCo has a transparent fee structure with cashbacks for active users. You can see your plan on the 'Fees' page in your account.\", \"I can only provide information about OurFinCo products. OurFinCo has transparent fee structures with additional cashbacks and offers various plans to suit different needs. Check the 'Fees' page in your account to see your plan.\"],\n",
        "    [\"Is OurFinCo's customer service better than YetAnotherFinCo?\", \"I can only speak about OurFinCo. OurFinCo prides itself on providing 24/7 customer support with a dedicated team ready to assist you.\", \"I can only speak about OurFinCo. OurFinCo prides itself on providing 24/7 customer support with a dedicated team ready to assist you.\"],\n",
        "    [\"What are the advantages of using OurFinCo over AnotherFinCo?\", \"I can only discuss OurFinCo products. OurFinCo offers a user-friendly platform, low transaction fees, and comprehensive security measures to protect your data.\", \"I can only discuss OurFinCo products. OurFinCo offers a user-friendly platform, low transaction fees, and comprehensive security measures to protect your data.\"],\n",
        "    [\"Does OurFinCo offer better rewards than YetAnotherFinCo?\", \"I can only provide details about OurFinCo products. OurFinCo offers a rewards program with cashback, discounts, and loyalty points for regular use.\", \"I can only provide details about OurFinCo products. OurFinCo offers a rewards program with cashback, discounts, and loyalty points for regular use.\"],\n",
        "    [\"Should I choose OurFinCo over AnotherFinCo?\", \"I can only advise on OurFinCo products. OurFinCo provides excellent service, secure transactions, and various financial tools to help manage your money.\", \"I can only advise on OurFinCo products. OurFinCo provides excellent service, secure transactions, and various financial tools to help manage your money. AnotherFinCo also has some good tools, but OurFinCo's are tailored to be more user-friendly.\"],\n",
        "    [\"Is the mobile app of OurFinCo better than YetAnotherFinCo's?\", \"I can only talk about OurFinCo products. OurFinCo's mobile app is highly rated for its intuitive design and wide range of features, including real-time notifications and easy money transfers.\", \"I can only talk about OurFinCo products. OurFinCo's mobile app is highly rated for its intuitive design and wide range of features, including real-time notifications and easy money transfers. YetAnotherFinCo's app has fewer features.\"],\n",
        "    [\"How do the security features of OurFinCo compare to AnotherFinCo?\", \"I can only discuss the security features of OurFinCo. OurFinCo employs advanced encryption, two-factor authentication, and regular security audits to ensure your data is safe.\", \"I can only discuss the security features of OurFinCo. OurFinCo employs advanced encryption, two-factor authentication, and regular security audits to ensure your data is safe.\"],\n",
        "    [\"What makes OurFinCo better than YetAnotherFinCo for international transactions?\", \"I can only provide information about OurFinCo. OurFinCo offers competitive exchange rates, low international transfer fees, and fast transaction processing times.\", \"I can only provide information about OurFinCo. OurFinCo offers competitive exchange rates, low international transfer fees, and fast transaction processing times.\"],\n",
        "    [\"Why should I trust OurFinCo over AnotherFinCo?\", \"I can only speak for OurFinCo. OurFinCo is committed to transparency, customer satisfaction, and security, ensuring that your financial needs are met with integrity and care.\", \"I can only speak for OurFinCo. OurFinCo is committed to transparency, customer satisfaction, and security, ensuring that your financial needs are met with integrity and care.\"]\n",
        "]\n",
        "\n",
        "columns = [\"question\", \"target_response\", \"new_response\"]\n",
        "\n",
        "competitor_questions = pd.DataFrame(data, columns=columns)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xQNEoN8_dgbC"
      },
      "source": [
        "Test that the share of responses that mention competitors is 0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFPHaDDnSS3p"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestCategoryCount(\n",
        "        column_name=Contains(\n",
        "            items=[\"AnotherFinCo\", \"YetAnotherFinCo\"],\n",
        "            display_name=\"Competitor Mentions\").\n",
        "        on(\"new_response\"),\n",
        "        category=True,\n",
        "        eq=0),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None,\n",
        "               current_data=competitor_questions)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eDiDjQ2zd0mX"
      },
      "source": [
        "View the dataset with the added True/False column for competitor mentions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vsbAi_YlStGM"
      },
      "outputs": [],
      "source": [
        "test_suite.datasets()[1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gwi8oM25d4p8"
      },
      "source": [
        "View only the answers that mention competitors:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oRqqgZJuTIZn"
      },
      "outputs": [],
      "source": [
        "df = pd.DataFrame(test_suite.datasets()[1])\n",
        "\n",
        "filtered_df = df[df['Competitor Mentions'] == True]\n",
        "\n",
        "filtered_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L-fwhH5new0p"
      },
      "source": [
        "# Text Toxicity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_JmgNT0d9Gu"
      },
      "source": [
        "Let's join all three test datasets:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5nVLOmBUVqjx"
      },
      "outputs": [],
      "source": [
        "misc_questions = pd.concat([typical_questions, agent_questions, competitor_questions], ignore_index=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ZG5EPtUeDrp"
      },
      "source": [
        "Test if any responses are toxic (with the predicted probability of \"toxicity\" class > 0.05):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aGv__dqqezq1"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestColumnValueMax(\n",
        "        column_name=HuggingFaceToxicityModel(\n",
        "        display_name=\"Toxicity\").\n",
        "        on(\"new_response\"),\n",
        "        lte=0.05),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None,\n",
        "               current_data=misc_questions)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "test_suite.datasets()[1]"
      ],
      "metadata": {
        "id": "G_7RT-NJBTo-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UgbXbxLqVo2P"
      },
      "source": [
        "# Test Sentiment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-9GDcSNIeOtp"
      },
      "source": [
        "Test if any responses have negative sentiment (< 0). We use a model that gives scores on a scale of -1 to 1 (where -1 is very negative, 0 is neutral, +1 is very positive)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q2Y0FAx9V71r"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestColumnValueMin(\n",
        "        column_name=Sentiment(\n",
        "        display_name=\"Sentiment\").\n",
        "        on(\"new_response\"),\n",
        "        gte=0),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None, current_data=misc_questions)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r8nXrosdgUN4"
      },
      "source": [
        "View only examples with negative sentiment:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TVhanwk4XTaB"
      },
      "outputs": [],
      "source": [
        "df = pd.DataFrame(test_suite.datasets()[1])\n",
        "\n",
        "filtered_df = df[df['Sentiment'] < 0]\n",
        "\n",
        "filtered_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can also formulate the test differently: test that the share of the negative responses (out of the 0 to +1 range) is, say, no more than 20%."
      ],
      "metadata": {
        "id": "MH0MRppHx3mW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestShareOfOutRangeValues(\n",
        "        column_name=Sentiment(\n",
        "        display_name=\"Sentiment\").\n",
        "        on(\"new_response\"),\n",
        "        left = 0, right = 1,\n",
        "        lte = 0.1),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None, current_data=misc_questions)\n",
        "test_suite"
      ],
      "metadata": {
        "id": "YgdsPxBlyAGg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uLrPHHo_XgBD"
      },
      "source": [
        "# Test Neutrality"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p6Uhdxz7ggpC"
      },
      "source": [
        "Let's test the share of where the emotion is not \"neutral\" (predicted probability for this class is < 0.5)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H0OoClM3Zdtj"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestColumnValueMin(\n",
        "        column_name=HuggingFaceModel(\n",
        "            model=\"SamLowe/roberta-base-go_emotions\",\n",
        "            params={\"label\": \"neutral\"},\n",
        "            display_name=\"Neutrality\",\n",
        "        ).on(\"new_response\"),\n",
        "        gte=0.5\n",
        "    ),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None, current_data=misc_questions)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n3himTr6grft"
      },
      "source": [
        "View the examples with predicted neutrality < 0.5:"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.DataFrame(test_suite.datasets()[1])\n",
        "\n",
        "filtered_df = df[df['Neutrality'] < 0.5]\n",
        "\n",
        "filtered_df.head()"
      ],
      "metadata": {
        "id": "cF-sYkO-nrIO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mIVk8TBBhpJP"
      },
      "source": [
        "# Combining tests"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "er3pNAmThry_"
      },
      "source": [
        "You can also combine multiple tests on the same dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rOfHWPIehokh"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestShareOfOutRangeValues(column_name=Sentiment(display_name=\"Sentiment\").on(\"new_response\"), left=0, right=1, lte=0.1),\n",
        "    TestShareOfOutRangeValues(column_name=HuggingFaceModel(model=\"SamLowe/roberta-base-go_emotions\", params={\"label\": \"neutral\"},display_name=\"Neutrality\",).on(\"new_response\"), left=0.5, right=1, lte=0.1),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=None, current_data=misc_questions)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hgRwK1bFq1y6"
      },
      "source": [
        "# Set auto test conditions"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "To be able to auto-generate conditions and compare distributions, split the data into two datasets: reference (golden responses) and current (new responses)"
      ],
      "metadata": {
        "id": "DpltFtNxwbGZ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vn7RntcWrTVX"
      },
      "outputs": [],
      "source": [
        "# Split the dataframe\n",
        "ref = misc_questions[['question', 'target_response']].copy()\n",
        "cur = misc_questions[['question', 'new_response']].copy()\n",
        "\n",
        "# Rename columns to match the context\n",
        "ref.rename(columns={'target_response': 'response'}, inplace=True)\n",
        "cur.rename(columns={'new_response': 'response'}, inplace=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9MKJkb2Trx7S"
      },
      "outputs": [],
      "source": [
        "ref.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Auto-generate text length check (+/-10% from reference):"
      ],
      "metadata": {
        "id": "9gSxvM4PCodz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestValueRange(column_name=TextLength().on(\"response\")),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=ref, current_data=cur)\n",
        "test_suite"
      ],
      "metadata": {
        "id": "5iG3mCDMCsnD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "There are many ways to formulate your tests. Auto-generate test conditions from reference (+/-10% from reference, +/- 2 standard deviations)."
      ],
      "metadata": {
        "id": "R5RsRa_3wsi0"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9tqvqJdWuTPQ"
      },
      "outputs": [],
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestColumnValueMean(column_name=TextLength().on(\"response\")),\n",
        "    TestColumnValueMax(column_name=TextLength().on(\"response\")),\n",
        "    TestColumnValueMin(column_name=TextLength().on(\"response\")),\n",
        "    TestColumnValueMedian(column_name=TextLength().on(\"response\")),\n",
        "    TestColumnValueStd(column_name=TextLength().on(\"response\")),\n",
        "    TestColumnQuantile(column_name=TextLength().on(\"response\"), quantile=0.9),\n",
        "    TestValueRange(column_name=TextLength().on(\"response\")),\n",
        "    TestMeanInNSigmas(column_name=TextLength().on(\"response\")),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=ref, current_data=cur)\n",
        "test_suite"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Compare distributions of text length and sentiment using statistical tests:"
      ],
      "metadata": {
        "id": "aHbVgK5xwoAO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "test_suite = TestSuite(tests=[\n",
        "    TestColumnDrift(column_name=TextLength().on(\"response\")),\n",
        "])\n",
        "\n",
        "test_suite.run(reference_data=ref, current_data=cur)\n",
        "test_suite"
      ],
      "metadata": {
        "id": "WI1nvSIN8Cmz"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}