DS103-Metrics-and-Data-Processing/DS103-L4-Survey-Design.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "50e6ddbb-52d3-4a02-9b69-100a4181e3b0",
   "metadata": {},
   "source": [
    "# DS103 Metrics and Data Processing : Lesson Four Survey Design"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1bd4bfc-9b15-4b80-87b4-b8b60eb56371",
   "metadata": {},
   "source": [
    "### Table of Contents <a class=\"anchor\" id=\"DS103L4_toc\"></a>\n",
    "\n",
    "* [Table of Contents](#DS103L4_toc)\n",
    "    * [Page 1 - Introduction](#DS103L4_page_1)\n",
    "    * [Page 2 - Survey Design Steps](#DS103L4_page_2)\n",
    "    * [Page 3 - Question Types](#DS103L4_page_3)\n",
    "    * [Page 4 - Sources of Survey Questions](#DS103L4_page_4)\n",
    "    * [Page 5 - Designing Survey Questions](#DS103L4_page_5)\n",
    "    * [Page 6 - Avoid Leading Questions](#DS103L4_page_6)\n",
    "    * [Page 7 - Survey Formatting Best Practices](#DS103L4_page_7)\n",
    "    * [Page 8 - Survey Review and Testing](#DS103L4_page_8)\n",
    "    * [Page 9 - Distributing Surveys](#DS103L4_page_9)\n",
    "    * [Page 10 - Using Survey Monkey](#DS103L4_page_10)\n",
    "    * [Page 11 - Key Terms](#DS103L4_page_11)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef5a8c65-ed48-4354-a257-e01b0fd733d4",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 1 - Introduction<a class=\"anchor\" id=\"DS103L4_page_1\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "af42bdd5-4656-485e-a896-19fda5750b46",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/388126031\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7ff73009e040>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Survey Design\n",
    "VimeoVideo('388126031', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b6ad526-95fa-43c4-bc43-1fe807708ed3",
   "metadata": {},
   "source": [
    "The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO103L04overview.zip)**.\n",
    "\n",
    "# Introduction\n",
    "\n",
    "While a lot of the time as a data scientist, you will get data handed to you, there will be times when you are asked to answer questions for which you have no data.  In those cases, it will be up to you to collect your own data, and often that is done with a survey.  In this module, you will learn about how to create and execute your own surveys.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7990ac7a-5a2f-4744-af14-1b7e385ea484",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 2 - Survey Design Steps<a class=\"anchor\" id=\"DS103L4_page_2\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612429df-e4ef-4d85-8460-5bc8914079c3",
   "metadata": {},
   "source": [
    "# What is a Survey?\n",
    "\n",
    "A survey is a way to gather information from others in which they are asked to provide answers to written questions.  The overarching goal of any survey is to translate a concept into some measurable data that can be studied and analyzed.  This presents many problems and can introduce a lot of biases, but with a solid foundation in survey design, you will hopefully be able to evade a lot of the common traps. \n",
    "\n",
    "---\n",
    "\n",
    "## Survey Design Steps\n",
    "\n",
    "Like anything else even somewhat creative, it pays to not rush in blindly, but rather to spend some time thinking through requirements and figuring about your overall goals.  Here are the general steps in survey development: \n",
    "\n",
    "* Determine the overall purpose of the survey\n",
    "* Brainstorm the questions you are trying to answer with the survey\n",
    "* Create survey items that align with your questions\n",
    "* Format the survey as appropriate for your questions and your audience\n",
    "* Write scoring and administration rules if needed\n",
    "* Test out your survey\n",
    "* Administer your survey\n",
    "\n",
    "---\n",
    "\n",
    "## Determine the Overall Purpose\n",
    "\n",
    "The overall purpose will typically be set by your supervisor or client.  It could be something broad, like \"See if employees are happy\" or it could be something a little more specific, such as \"See if there are any differences in employee satisfaction by department.\"  Either way, do your best to find out exactly what they're looking for, by asking as many questions as possible.  Especially if you have to start with something broad, you'll want to drill down to exactly what your supervisor or client wants.  In the case of \"see if employees are happy,\" you might start by asking how they would define happiness and if they want to look at any sub-categories of employees.  The answer to these questions will start driving you in the right direction.\n",
    "\n",
    "---\n",
    "\n",
    "## Brainstorm Questions to Answer\n",
    "\n",
    "Once you have a very broad, overall purpose, you'll want to get into even more specifics. What are the questions you absolutely MUST be able to answer with this survey when you're done? Typically these are called *evaluation questions* or *research questions*, and it's a good idea to have at least three, though they may have subparts to them.  For instance, if you are surveying caregivers of the elderly, one of your overarching evaluation questions might be \"What resources do caregivers need to best care for their care recipient?\" and you can answer that question by looking at physical, mental, and social needs. \n",
    "\n",
    "Take your time brainstorming these questions, and if possible, work with the supervisor or client who set your overarching goal to make sure you're getting at the things he or she cares about most.  These evaluation questions will guide all of the survey items you create, as well as your survey audience.\n",
    "\n",
    "In the next few pages, you will learn about the next step in the process, which is to create survey items that align with your questions.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832358fd-55fa-420a-8aa2-46fbf2097986",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 3 - Question Types<a class=\"anchor\" id=\"DS103L4_page_3\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f564e3e7-3a7f-4936-b3a3-2e0a9ab78512",
   "metadata": {},
   "source": [
    "# Question Types\n",
    "\n",
    "There are many different ways you can ask questions on a survey. Some will work better than others for your intended purpose, so it's a good idea to understand the different types so you can make the best use of your survey!\n",
    "\n",
    "---\n",
    "\n",
    "## Multiple Choice\n",
    "\n",
    "You are probably pretty familiar with the multiple choice question, since it is the item of choice on many a school exam.  Here is an example of how a multiple choice question could be used on a survey: \n",
    "\n",
    "```text\n",
    "What is your approximate yearly income? \n",
    "-Less than $15,000\n",
    "-$15,00-$29,999\n",
    "-$30,000-$59,999\n",
    "-$60,000-$89,999\n",
    "-Above $90,000\n",
    "```\n",
    "\n",
    "With a multiple choice question, there is only one correct answer, which means that all the options must be exclusive - you shouldn't have any survey respondent who fits in more than one income category, for example. Multiple choice questions provide you with categorical data.\n",
    "\n",
    "---\n",
    "\n",
    "## Select All that Apply \n",
    "\n",
    "A *select all* question is very similar to a multiple choice question, but instead of having only one correct answer, there can be many. These questions typically start or end with the phrase \"Please select all that apply,\" which is how they have earned their name. Here is an example of a select all question: \n",
    "\n",
    "```text\n",
    "Which of the following items did you spend money on in the last month? Please select all that apply.\n",
    "[] Gas\n",
    "[] Groceries\n",
    "[] Rent / Mortgage\n",
    "[] Eating out\n",
    "[] Hobbies\n",
    "[] Clothes\n",
    "[] Travel\n",
    "[] Utilities\n",
    "[] Household goods\n",
    "```\n",
    "\n",
    "Just because someone has spent money on gas in the last month does not mean that he or she could not have spent money on any of the other categories, such as rent or groceries.  Therefore a select all question is much more appropriate than a multiple choice question in this scenario, as select all questions are not exclusive, like multiple choice questions are. The data you receive from a select all question is also categorical, but beware! You will not be able to calculate frequencies for the question as a whole, like you might for a multiple choice question, but instead, will have frequencies for each answer choice, since otherwise you would end up with a number of responses that is much greater than the number of people who actually answered the survey!\n",
    "\n",
    "---\n",
    "\n",
    "## Scale\n",
    "\n",
    "Often, surveys are used to measure subjective things such as attitudes. Sometimes, attitudes are measured with multiple choice questions that have responses like \"agree\" and \"disagree,\" but they can often be done as a scale question.  Scale questions are laid out horizontally and typically have some sort of scale or ruler system to mark out the different response options, like this: \n",
    "\n",
    "![A statement that reads, my manager considers my feelings. To the right of the statement is a five point rating scale. The point on the left is labeled strongly agree. The point in the middle is labeled neutral. The point on the right is labeled strongly agree.](Media/survey2.png)\n",
    "\n",
    "The benefit of a scale question over a multiple choice question when asking about subjective information is that it is more visual, and it takes up less vertical space.  However, scale questions are not as good for those taking surveys on mobile devices such as a phone or tablet, since it can be hard to see all the options at once.\n",
    "\n",
    "Scale questions also provide you with categorical data.\n",
    "\n",
    "---\n",
    "\n",
    "## Matrix\n",
    "\n",
    "A matrix question is very similar to a scale question, but now, there are more than one! Matrix questions can look like the one below, or they may start with a common question stem, such as: \"In the workplace, I feel that...\"\n",
    "\n",
    "![Statement one, my manager considers my feelings. Statement two, my wellbeing is a priority for my manager. Statement three, my manager defends me from unfair criticism. To the right of each statement is a five point rating system. From left to right, the points are labeled strongly disagree, disagree, neutral, agree, and strongly agree.](Media/survey3.png)\n",
    "\n",
    "Like scale questions, the major benefit of matrix questions is that they save a lot of vertical space.  Another advantage of matrix questions is that they increase organization on your survey, reduce the amount of repetition, and generally help reduce confusion.  However, as with scale questions, matrix questions may not read well on mobile devices.\n",
    "\n",
    "Scale questions provide you with categorical data only.\n",
    "\n",
    "---\n",
    "\n",
    "## Write-in Response\n",
    "\n",
    "Write-in responses are not used as often as some of the other survey questions above you have learned about; however, they are one of the only ways you can be provided with continuous data, which adds value! For instance, you could have a write in question about age, instead of using a multiple choice question about age categories, like this: \n",
    "\n",
    "```text\n",
    "What is your age? [             ]\n",
    "```\n",
    "\n",
    "Write in questions do not always yield continuous data, however; they often will yield qualitative, unorganized string responses instead.  For example, you could ask this: \n",
    "\n",
    "```text\n",
    "What is your favorite food? [          ]\n",
    "```\n",
    "\n",
    "And you should be prepared for all chaos to break loose! Write-in responses are a good way to gather continuous data when you need it, and they are also good to employ if you don't know a lot about your target subject or audience and are worried that there are things \"you don't know you don't know.\" It allows your survey respondents to have a little more control over their answers, and you may glean some valuable information you would have missed otherwise if you had restricted their responses in a structured question.  However, the data wrangling that will come with a write-in response can be nightmarish! Even for a simple question about age, unless you put character limits and data type limits on it, you may end up with people typing out numbers, writing silly things like \"60 years young,\" or making typos like \"1111 years old,\" which can't possibly be correct.  So you'll want to weigh the pros and cons of write-in responses very carefully!\n",
    "\n",
    "For this reason, many data scientists will include write-in response boxes as part of select all that apply questions that say \"Other,\" and allow folks the ability to write in a response only if they don't think the others are adequate and inclusive of their situation.  This limits the amount of data you'll need to sift through, but still allow some question structure.\n",
    "\n",
    "Write-in responses can provide continuous data or qualitative data, that can be coded into categorical data if needed and possible.\n",
    "\n",
    "---\n",
    "\n",
    "## Open-Ended Essay\n",
    "\n",
    "Open-ended questions are very similar to write-ins, but they allow for much longer responses.  Typically, they will ask deeper questions that require an expansive response.  An example of an open-ended essay question might be: \n",
    "\n",
    "```text\n",
    "Is there anything else you would like us to know about your experience with Data Tech?\n",
    "```\n",
    "\n",
    "As might be expected, analyzing these kinds of questions can be a nightmare, and really requires in-depth natural language processing or qualitative analysis to get meaningful data from these questions.  Therefore, they are typically used sparingly.  However, even if you know you won't have the manpower to analyze absolutely everything, including one open-ended question at the end can make the respondent feel included and yield some surprising information you might not otherwise have gained.\n",
    "\n",
    "Open-ended essay questions only provide you with qualitative data, and it is very difficult to code it into categorical data.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1aade2-59af-4cbc-b5ed-da31e39f266d",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 4 - Sources of Survey Questions <a class=\"anchor\" id=\"DS103L4_page_4\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba2ed6e-98dd-43f7-8761-6ecc9a13af08",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# Sources of Survey Questions\n",
    "\n",
    "Now that you've learned about the different types of survey questions, you may be overwhelmed at the prospect at writing all those questions up yourself.  But never fear! You can often borrow survey questions from other places legally, and not only is this not considered cheating, but it is encouraged! Don't re-invent the wheel - there's a good chance that someone else has not only created the survey question you are looking for, but has done it far better than you ever could have.  There is a process called *survey validation* that ensures that your survey will be as unbiased as possible, and whenever possible, you want to look for validated survey items so that you don't have to go through this process yourself, as it is quite time consuming and mathematically advanced.  \n",
    "\n",
    "Where do you find these miraculous validated survey questions? By and large, it will take a little bit of work and some digging, but it is worth it, because you'll be learning a lot about your topic along the way.  For a few quick wins, check out these resources: \n",
    "\n",
    "* **[UK Data Service Variable and Question Bank](https://discover.ukdataservice.ac.uk/variables)** : This website allows you to search for survey questions by keyword. \n",
    "* **[Constant Contact Sample Survey Questions, Answers and Tips](https://www.constantcontact.com/aka/docs/pdf/survey_sample_qa_tips.pdf)** : This PDF provides survey question suggestions about basic business topics, such as satisfaction with customer service, events, membership, and websites. \n",
    "* **[Wikimedia Learning and Evaluation Question Bank](https://meta.wikimedia.org/wiki/Learning_and_Evaluation/Question_Bank)**: This question bank is excellent for demographics and program participation questions.  If you're doing data science for education, you'll be able to use other sections from it as well.\n",
    "* **[SurveyLegend](https://www.surveylegend.com/likert-type-scale-responses-examples-with-examples/)**: This website provides you with a wide variety of scaled response options for questions, commonly called *Likert Scales*. \n",
    "\n",
    "---\n",
    "\n",
    "## Peer-Reviewed Articles\n",
    "\n",
    "You can also search for peer-reviewed articles that have included survey questions, and these are often the best source of survey questions, because you can guarantee that they have been thoroughly reviewed, not just by the authors of the article, but by the scientific community at large.  Often, academic researchers will take years to develop their surveys and ensure they are properly validated, so if you have the time to devote to finding them, and your company values rigor, then searching for research articles with the right surveys is absolutely the way to go.\n",
    "\n",
    "---\n",
    "\n",
    "### Where Do you Find Peer-Reviewed Articles? \n",
    "\n",
    "If you can link into any large online library system, that is the best way to find peer-reviewed articles.  You may be able to get access through your public library, your new workplace, through a previous education experience, or maybe even through a family member going to school that is willing to share their library time.  If you do have access to an online library system, then some of the bigger databases you'll want to search in for validated surveys are as follows: \n",
    "\n",
    "* **Academic Search Complete:** Good general topic search database\n",
    "* **Business Source Complete:** For business topics\n",
    "* **CINAHL:** For healthcare\n",
    "* **Cochrane Library:** Healthcare\n",
    "* **Education Database:** For education\n",
    "* **Health and Psychosocial Instruments (HaPI):** Provides surveys for health, business, and psychology\n",
    "* **JSTOR: The Scholarly Journal Archive:** General topic search database\n",
    "* **MEDLINE:** Healthcare\n",
    "* **Mental Measurements Yearbook:** Provides validated surveys for psychology and human interest topics\n",
    "* **PsycINFO:** Articles on behavioral sciences and mental health\n",
    "* **ScienceDirect:** General purpose science database\n",
    "\n",
    "In your particular area, there will definitely be additional databases that are helpful and interesting.  Use your own subject matter expertise and good judgement to navigate and find those!\n",
    "\n",
    "You may be wondering why so many of the databases above are psychological in nature.  This is because a lot of the topics covered in these psychology databases are broad and apply to many other topics as well, such as business and manufacturing (industrial/organizational psychology), human nature (personality, behavior, motivation, etc.), health (health and medical psychology), education (educational and cognitive psychology), and user experiences (neuroscience), to name just a few, all have their backbone in psychology. So don't discount the resource immediately just because it says \"psychology!\"\n",
    "\n",
    "---\n",
    "\n",
    "### Google Scholar\n",
    "\n",
    "If you don't have access to a large library system, then you can make do with Google Scholar. It is a free search service for research articles, and there are many articles available for free as well, but it won't have nearly as many as a library you hook into.  It also doesn't operate as well as a search service, so it can be more difficult to find what you want. Nevertheless, you'll learn how to use Google Scholar in case it is your only option!\n",
    "\n",
    "Oddly enough, Google makes Google Scholar hard to find.  To do so, you'll actually need to search for Google Scholar on Google, like this:\n",
    "\n",
    "![A web broswer page. Google scholar has been entered into the google search bar. Below the searchbar is the link to google scholar and links to specific pages within the google scholar website.](Media/survey4.png)\n",
    "\n",
    "Once you click on the very first link, you will be into Google Scholar proper, which looks like this: \n",
    "\n",
    "![A web browser page that has loaded the U R L, H T T P S colon forward slash forward slash scholar dot google dot come. The page title reads google scholar. A search bar is below the title.](Media/survey5.png)\n",
    "\n",
    "Now, you can always type in stuff into the general large box up above, but often it just is not specific enough.  And unlike most other database searches, Google Scholar does not respond to typical search specifiers, such as putting things in quotes or use of boolean operators. Therefore, you'll need to make use of the ```Advanced search``` option for anything remotely complex.  Here's how to access it: just click on the three lines in the top left of the screen, and then this menu will appear: \n",
    "\n",
    "![A web broswer showing the sidebar navigation of the google scholar website. There are links for my profile, my library, alerts, metrics, advanced search, and settings. Advanced search is highlighted.](Media/survey6.png)\n",
    "\n",
    "Once you click on ```Advanced search```, you'll get these options:\n",
    "\n",
    "![The google scholar advanced search window, which contains fields allowing the user to narrow his or her search.](Media/survey7.png)\n",
    "\n",
    "Here, if you need to search for an exact phrase, you can do so, you can search using the boolean operator NOT under \"without the words\" and you can search for specific authors, journals, or dates, all of which can really help you narrow down a search if you need it. \n",
    "\n",
    "For learning purposes, you'll search for the terms \"disability questionnaire.\" If you're looking on Google Scholar for survey items, always remember to not only include your topic but also the term survey or questionnaire, etc.  so that it will not only find you the topic but also some questions you can actually use. You may need to play around with different terms and synonyms to get the results you want. Here are the search results: \n",
    "\n",
    "![Google scholar search results from searching the term disability questionnaire.](Media/survey8.png)\n",
    "\n",
    "As you can see, those that are available show up in the right hand side with either a PDF or a link to the article. Not all articles will be available, and not all will contain the actual survey questions they used, either.  So it will take some looking around to find what you need. Often surveys are included as appendices at the end of an article, or they may be included in the ```Methods``` section of an article.  Looking closer at the second result down, if you bring up the PDF and scroll towards the end, you will see that the original questionnaire is indeed included.  You can use these questions, but only if you cite them on your survey. \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Without citing the source of survey questions, this is plagarism! Besides being illegal and immoral, plagarism could get your company in trouble, so always cite your sources to stay above board!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ead4ad3c-2fe6-457e-8924-1410e951bebd",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 5 - Designing Survey Questions<a class=\"anchor\" id=\"DS103L4_page_5\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbbc06a2-e702-47a1-942f-cbcbe31221a7",
   "metadata": {},
   "source": [
    "\n",
    "# Designing Survey Questions\n",
    "\n",
    "If after careful searching, you can't find what you want or need, you will need to create your own survey questions. There are some definite rules and best practices when it comes to creating your own survey questions, so focus up! You'll learn best practices for: \n",
    "\n",
    "* Questions\n",
    "* Answers\n",
    "* Things to avoid\n",
    "\n",
    "---\n",
    "\n",
    "## Phrase All Items as Questions\n",
    "\n",
    "All items on your survey should be phrased as a question, so that there is no confusion! For instance, if you want to know someone's weight, ask \"What is your current weight in pounds?\" and leave a write-in response blank, rather than just putting \"Weight:\" with a write-in response blank.  This helps alleviate confusion and makes things much more specific.  you don't want to leave it up to their interpretation, or you might get some odd data.  Maybe they include the weight they'd like to be.  Maybe they include the weight of their pet or child.  You never know what is going on inside a survey participant's head!\n",
    "\n",
    "---\n",
    "\n",
    "## Don't Give the Answer You're Looking For in the Question\n",
    "\n",
    "You'd be surprised how easy it is to accidentally give away the answer you're looking for in the question text. For instance, what if you asked this question: \n",
    "\n",
    "```text\n",
    "Are you aware that data science is a new field?\n",
    "-Yes\n",
    "-No\n",
    "```\n",
    "\n",
    "By asking this question, you are telling your participants that data science is a new field! So a much better question would be something like this one: \n",
    "\n",
    "```text\n",
    "How did you first learn that data science is a new field? \n",
    "-From a friend\n",
    "-From a coworker\n",
    "-Social media\n",
    "-I had not heard of data science before taking this survey\n",
    "```\n",
    "\n",
    "Now you have your bases covered, and you are learning much more from the participant, still with a single question.\n",
    "\n",
    "---\n",
    "\n",
    "## Don't Make Them Do Math\n",
    "\n",
    "Anytime you're asking a survey respondent to calculate something is usually a no-go! At best, it is time-consuming and complicated.  They may or may not actually do what you asked.  At worst, they will make a mistake in their calculations and you will end up with inaccurate data.  For instance, don't ask someone what their body mass index is.  That's not a figure anyone is likely to have memorized, and they probably won't know the formula for calculating it.  It's much better to ask for their height and weight, which is something that most do know, and then calculate BMI later.  You'll be able to calculate it easily all down the column using Python and R, and will save them much confusion, muddling, and wrong-answers.\n",
    "\n",
    "Here's another example: instead of asking for their distance away from your particular company, you could just ask for their address instead and put it into Google Maps later to find out the distance in miles.  Although time consuming, it is bound to be much accurate!\n",
    "\n",
    "---\n",
    "\n",
    "## Be Specific\n",
    "\n",
    "Specificity is key! You've already hit on that a little with defining units, but it also works for defining terms and being clear in your question wording. \n",
    "\n",
    "What if you asked someone not very familiar with data scientist how often they did data wrangling? They might be scratching their heads forever, and you probably wouldn't get great data out of it.  This would be a much better option: \n",
    "\n",
    "```text\n",
    "How often do you perform data wrangling skills such as adding columns, flipping data, subsetting data, splitting data, or recoding data? \n",
    "```\n",
    "\n",
    "Now participants will have a much easier time understanding what you are asking and you have defined it nicely.  You could even ask about each example individually if it added value towards your overall survey goals.\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Survey generation software will often give you an option to add an explanation about something when participants hover over the word, so you can utilize this feature to better define things as well.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Provide Time Frames\n",
    "\n",
    "Your metrics need to be time-bound, and your survey questions should be too.  Research has shown that memory recall is really only accurate going back two weeks in history, so try to keep a reporting window short.  Asking someone how many times they downloaded data in the last year just won't work! As a data scientist, you probably have downloaded a lot of data, and it's such a basic thing that you don't keep track of it very well, let alone over the course of a year! But if you can ask how many times they downloaded data that day or that week, it will be much easier to remember and thus you will increase the accuracy of your responses.\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded.\n",
    "\n",
    "```c-lms\n",
    "start-activity: Designing Questions Quiz\n",
    "```\n",
    "\n",
    "```c-lms\n",
    "topic: Designing Survey Answers\n",
    "```\n",
    "\n",
    "# Designing Survey Answers\n",
    "\n",
    "There are just as many things to take into consideration when designing survey answers, also called *responses*, as there are when writing questions.  \n",
    "\n",
    "---\n",
    "\n",
    "## Provide An Exhaustive List of Answers\n",
    "\n",
    "Ever take a survey and end up frustrated, because there was no way you could appropriately express yourself? Maybe the response options didn't accurately reflect your experiences, or there was no place to write in an \"other\" experience? That scenario is exactly what you want to avoid in survey creation! So it is important to include an exhaustive list of options.  Here's an example of what NOT to do:\n",
    "\n",
    "```text\n",
    "What do you want to do with your hair?\n",
    "-Make it longer\n",
    "-Make it shorter\n",
    "```\n",
    "\n",
    "In the example above, there are only two options listed for what someone could do with their hair: they could make it longer or shorter. But what if they liked the length of their hair, and wanted to keep it the same? What if they don't pay any attention to their hair and don't have any particular hair-related goals? Then the respondent would be left feeling very frustrated! A better question, with more response options, might be something like this: \n",
    "\n",
    "```text\n",
    "What do you want to do with your hair?\n",
    "-Make it longer\n",
    "-Make it shorter\n",
    "-Maintain my hair length\n",
    "-I am not trying to do anything with my hair\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Ensure Answers are Exclusive\n",
    "\n",
    "Especially when you ask about categories of something, it becomes easy to include answers that overlap accidentally.  Think about this example: \n",
    "\n",
    "```text\n",
    "How old are you? \n",
    "-0-25 years old\n",
    "-25-40 years old\n",
    "-40-55 years old\n",
    "-55-70 years old\n",
    "-Older than 70\n",
    "```\n",
    "\n",
    "See anything wrong with it? How about if you're aged 25, 40, or 55? Which option should you choose? You would fall into two categories, and thus when it came time for analysis, you wouldn't be able to confidently say how many people were in each age group.  A much better option would be something like this:\n",
    "\n",
    "```text\n",
    "How old are you? \n",
    "-0-25 years old\n",
    "-26-40 years old\n",
    "-41-55 years old\n",
    "-56-70 years old\n",
    "-Older than 71\n",
    "```\n",
    "\n",
    "Now, there is only one option for each person, and all the answers are exclusive.\n",
    "\n",
    "---\n",
    "\n",
    "## Make Units Clear\n",
    "\n",
    "It's also incredibly important to make the units you are asking about clear! What if you were asked how long you'd be in a particular job, like this: \n",
    "\n",
    "```text\n",
    "How long have you been with DataLove Corporation? \n",
    "-0-6\n",
    "-7-12\n",
    "-13-18\n",
    "-19-24\n",
    "```\n",
    "\n",
    "Without any specification on there, that question could easily be interpreted as either months or years.  To get the answers you're looking for, and not bias your data, it's important to specify the unit here.  Correcting that question would look like this: \n",
    "\n",
    "```text\n",
    "How long have you been with DataLove Corporation? \n",
    "-0-6 months\n",
    "-7-12 months\n",
    "-13-18 months\n",
    "-19-24 months\n",
    "```\n",
    "\n",
    "Since you're building skills on top of each other, you'll notice that none of these categories overlap, either, so the answer choices remain exclusive.\n",
    "\n",
    "---\n",
    "\n",
    "## Scale Responses Should be Balanced\n",
    "\n",
    "When using scales, it's important that the response options are balanced and have equal intervals between the choices.  For instance, something like like is a good choice: \n",
    "\n",
    "```text\n",
    "Strongly agree\n",
    "Agree\n",
    "Disagree\n",
    "Strongly disagree\n",
    "```\n",
    "\n",
    "The above works nicely because you have balance - there is a strongly agree at the top, and a strongly agree at the bottom, and agree to counteract disagree in the middle.  This means that a respondent isn't influenced by verbal clues to lean their response one way or the other.  But consider something like this: \n",
    "\n",
    "```text\n",
    "Strongly agree\n",
    "Agree\n",
    "Somewhat agree\n",
    "Disagree\n",
    "```\n",
    "\n",
    "Notice how three of the four options are now positive and in some form of agreement? This unconsciously leads the participant to pick one of the top three options, which of course can bias your results.\n",
    "\n",
    "---\n",
    "\n",
    "## Use Ranges When Asking About Sensitive Things\n",
    "\n",
    "Sometimes folks can be put off by having to write out specific information that is sensitive in nature.  For instance, salary, age, or household income may all be something you squirm about when discussing. It becomes a more manageable and approachable topic when it's in ranges.  You don't feel quite so singled out.  However, you might not be able to get that continuous response you were hoping for; so it's a balancing act. Instead of just asking outright something like \"What is your household income?\" you may instead want to ask the following: \n",
    "\n",
    "```text\n",
    "Into what range does your current household income fall?\n",
    "-Less than $25,000\n",
    "-$25,000-$49,999\n",
    "-$50,000-$74,999\n",
    "-$75,000 or more\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Think Carefully About \"I Don't Know\" and \"Neutral\" Response Options\n",
    "\n",
    "If you allow humans to waffle, they will - it's in their nature! So typically, you won't want to use \"I don't know\" as an answer choice, or use a neutral option in your response scale.  It's better to have them pick a side, even if that side is only \"slightly agree\" or \"slightly disagree.\"  If you do include a middle choice - be aware of the ramifications.  How impressed is your boss going to be to hear that over half of participants felt neutrally about a particular topic? Have you learned anything useful that can guide actions? Probably not.\n",
    "\n",
    "The only time you may want to include a \"get out of jail free card\" is when a survey is mandatory.  As a general rule, a survey SHOULD NOT BE MANDATORY! It goes against ethics and human rights to require that someone participate in a survey, but in a few situations, like when working with Army commanders, there may be extenuating circumstances that require everyone in a unit to participate. The Army calls this \"voluntold.\" If that's the case, you want to ensure that answering all the questions on the survey is not mandatory, and if participants don't feel comfortable answering, they don't have to; or if they do have to answer, they can choose a neutral option or a \"I don't know\" option so they are not forced to share personal information if they don't want to.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9e3b8a-9224-447f-8cf7-5035cae86421",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 6 - Avoid Leading Questions<a class=\"anchor\" id=\"DS103L4_page_6\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a81e2098-28ef-4ff5-a155-afe824de4323",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "## Avoid Leading Questions\n",
    "\n",
    "A leading question is a question in which you have accidentally biased the respondent towards answering in a certain way.  For instance, if you asked, \"To what extent do you agree with the following statement,\" you've only said the word agreed, not disagreed, so respondents are going to be slightly biased towards agreeing.  A better way to phrase something like that would be: \"To what extent do you agree or disagree with the following statement.\" \n",
    "\n",
    "You often see leading questions in political debates and the like.  You'll hear questions that start with \"Wouldn't you agree that...?\" or \"Don't you dislike...?\" Unless you're really thinking on your feet and ready to be assertive, sometimes it's easy to just say \"Yes\" to whatever the question might be. \n",
    "\n",
    "---\n",
    "\n",
    "## Avoid Double Barreled Questions\n",
    "\n",
    "A double-barreled question is one in which you ask about two different things in the same question. For instance, the question below asks: \n",
    "\n",
    "```text\n",
    "Do you believe data scientists should be paid more for working remotely and for working with big data?\n",
    "```\n",
    "\n",
    "That covers two items: working remotely, and working with big data.  A participant could feel strongly about one part of that, but not the other, and then they would be unsure of how to answer this question.  Chances are, the participant would make some sort of compromise, and you wouldn't get a true answer about either.  This would be much better as two separate questions, as follows: \n",
    "\n",
    "```text\n",
    "Do you believe data scientists should be paid more for working remotely?\n",
    "-Yes\n",
    "-No\n",
    "\n",
    "Do you believe data scientists should be paid more for working with big data?\n",
    "-Yes\n",
    "-No\n",
    "```\n",
    "\n",
    "--- \n",
    "\n",
    "## Avoid Value Laden Terms\n",
    "\n",
    "A value-laden term is anything that casts subtly casts judgement.  Words like \"good\", \"bad,\" or \"alright\" are all value-laden terms.  They aren't specific enough to do any good in a survey question, but they can isolate people.  What if you eat at your desk regularly because you get a short lunch break, but encounter the survey item down below? \n",
    "\n",
    "```text\n",
    "Please rate your agreement or disagreement with the following statement: \n",
    "\"People who eat at their desk are disgusting crumb monsters.\"\n",
    "```\n",
    "\n",
    "How are you going to feel about that survey item? You'll probably react pretty negatively, especially if you are a neat eater and tidy up after yourself every day.  You may be so angry you stop taking the survey altogether.  You may continue with the survey, but see so much red that you answer all the questions differently than you might normally.  All of that emotion is for nothing, too, because using the term \"disgusting crumb monster\" doesn't actually provide a lot of specific information. You can't probably do a lot with the responses from that question.  But changing it to something like the below would provide useful information and hopefully not offend any participants: \n",
    "\n",
    "```text\n",
    "Please rate your agreement or disagreement with the following statement:\n",
    "\"It is easy to accidentally get food stuck in my keyboard when I eat at my desk.\"\n",
    "```\n",
    "\n",
    "Now you may actually be able to collect useful information about a topic, and there's no swirl of emotion around the question.\n",
    "\n",
    "---\n",
    "\n",
    "## Avoid Double Negatives\n",
    "\n",
    "Don't you think you should not do this? \n",
    "\n",
    "Joking aside, a double negative means that you have two negative statements or \"not\" statements in a sentence, and they cancel each other out to mean the opposite.  They can be hard to think through and answers can easily become incorrect.  Double negatives located in everyday life are tricky; double negatives in a survey with agree/disagree statements can be disastrous! \n",
    "\n",
    "Try this bad boy out: \n",
    "\n",
    "```text\n",
    "To what extent do you agree or disagree with the following statement?\n",
    "\"I do not think that data privacy is unimportant.\" \n",
    "\n",
    "-Strongly agree\n",
    "-Agree\n",
    "-Disagree\n",
    "-Strongly disagree\n",
    "```\n",
    "\n",
    "Wahoo! You have to figure out what the statement means before you can figure out if you agree with it or not. This statement means that data privacy is important, because the double negatives cancel each other out.  So, hopefully y'all would chose strongly agree.  But think about how much head-scratching that took! It would be much easier to do something like this: \n",
    "\n",
    "```text\n",
    "To what extent do you agree or disagree with the following statement?\n",
    "\"Data privacy is important.\" \n",
    "\n",
    "-Strongly agree\n",
    "-Agree\n",
    "-Disagree\n",
    "-Strongly disagree\n",
    "```\n",
    "\n",
    "Nice and clean and simple now, yes?\n",
    "\n",
    "Since you're reinforcing your previous learning, note that the response options for this question are once again nice and balanced!\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42cb7d8a-aced-42e5-ac24-32acf8c95473",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 7 - Survey Formatting Best Practices<a class=\"anchor\" id=\"DS103L4_page_7\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf0debaf-5fae-4035-8e7d-ae62e62456cb",
   "metadata": {},
   "source": [
    "# Survey Formatting Best Practices\n",
    "\n",
    "Along with writing your questions properly, there are also a set of best practices for how the survey as a whole should be formatted. \n",
    "\n",
    "---\n",
    "\n",
    "## Survey Length\n",
    "\n",
    "In an ideal world, your survey should take about five minutes for someone to complete (about 6-10 questions). However, that may be impossible to manage, depending on your goals and upon the wishes of meddling bosses and clients.  Do your best to keep the survey length under 15 minutes, or you'll have a lot of people stop halfway through or plain old not start your survey in the first place.\n",
    "\n",
    "---\n",
    "\n",
    "## General Design Principles\n",
    "\n",
    "The same basic design principles that you've learned about data visualizations and infographics also apply here.  You'll want to keep to no more than 2 or 3 colors and fonts, and avoid neon colors at all costs! The font size should be readable, with it neither being too small (hard to read on a large desktop monitor) nor too large (hard to scroll through on a mobile phone). It should be easy to differentiate the sections of the survey with headers at the top of the page, and you should be able to tell the questions from their answers.\n",
    "\n",
    "---\n",
    "\n",
    "## Reading Level\n",
    "\n",
    "A general rule of thumb is that your survey should be written at a 6th grade reading level.  This means no large, obnoxious long words, and include definitions for things even remotely complex.  You can check the reading level in MS Word by going to ```File -> Options -> Proofing -> Show readability statistics```, and then run a spelling and grammar check.  But there seems to be a few bugs in the current version, so if that's not working for you, or if you don't have access to Microsoft products, there are also multiple websites dedicated to checking reading level online.  Here's one: **[the Readability Test Tool](https://www.webfx.com/tools/read-able/check.php)**.  Simply choose the option to ```TEST BY DIRECT INPUT``` and then copy and paste your text into the box above and press the ```CALCULATE READABILITY``` button. \n",
    "\n",
    "![The readability test tool, a website that checks a texts reading level. The website allows you to enter or paste text by direct input. You can also test text by U R L or by referer.](Media/survey9.png)\n",
    "\n",
    "You get access to many different statistics as a result, but the most important thing to note is the average grade level shown in the white ```TEST RESULTS``` box, as well as the ```Flesch Kincaid Reading Ease``` score.  This score ranges from 0-100, with 0 being the most difficult to read, at a college graudate level, and 100 being the easiest to read, at a fifth grade reading level.  Since you are aiming for a 6th grade reading level or so for surveys issued to the general public, you are looking for a score of above 80. Clearly the example text run through here would not be appropriate for surveys!\n",
    "\n",
    "![Test results after text is analyzed by the readability test tool, a website that checks a texts reading level. The test results summar reads, your text has an average grade level of about eleven. It should be easily understood by sixteen to seventeen year olds. Readability indices include flesch kincaid reading ease, fifty six point seven, flesch kincaid grade level, nine point nine, gunning fog score, twelve point eight, smog index, nine point four, and coleman liau index, eleven point five.](Media/survey10.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Informed Consent Page\n",
    "\n",
    "Your company may require an ethical review to ensure that your survey will do no harm to participants.  That can be something formal, called the *Institutional Review Board*, or IRB, or it could be something a little less formal.  Whether an ethical review occurs or not, it is important to ensure that you get consent from all participants who are going to take your survey.  You want to make sure that everyone is a willing participant and knows that they don't have to take your survey and that there will be no negative repercussions if they choose not to take your survey.  In order to make sure they can consent properly, participants need information about what you're doing and why.  So it's important to provide all the information possible about your survey is as concise a manner as possible.  This is usually done with a short descriptive paragraph and then some frequently asked questions.  Here are some things to include on your informed consent page: \n",
    "\n",
    "* Basic information about your survey and its goals\n",
    "* Who to contact if participants have questions\n",
    "* What are the risks/harms that may come from taking the survey?\n",
    "* What are the benefits of taking the survey?\n",
    "* Will the participants' information be shared?\n",
    "* Can the participant stop taking the survey at any time?\n",
    "* Is participating in the survey voluntary?\n",
    "\n",
    "Once the participants have all the information they need to make a decision, then there needs to be some action that takes place that serves as their consent.  If you are doing a paper-and-pencil survey, then this action is a signature.  But in the case of an online survey, language like this is typical: \n",
    "\n",
    "```text\n",
    "I understand that my participation in this research study is entirely voluntary.  I may refuse to participate or stop at any time.  \n",
    "\n",
    "By clicking NEXT, you consent to take part in this study.\n",
    "```\n",
    "\n",
    "The participant may choose to stop there, but you know that if they move onto the next page with the survey, then they are willing and understand everything that taking the survey entails.\n",
    "\n",
    "---\n",
    "\n",
    "## Demographics\n",
    "\n",
    "Next, you'll most likely want to collect demographics from your participants - things like age, gender, race, income, location - whatever you think may play a role in the information you're gathering that you'd like to control for later in analyses. Remember that demographics can be sensitive to some people, so it is a good idea to ask them as ranges rather than a write-in response, with the possible exception of age, just because it is so valuable to have it as a continuous number rather than as a category.\n",
    "\n",
    "---\n",
    "\n",
    "## Concepts Should Flow\n",
    "\n",
    "Next, you'll want to include your regular survey questions, and it is important that they are nicely introduced and have a flow to them.  This not only makes it easier for the participants to understand what is being asked of them, but also increases their recall.  When things flow nicely and make logical sense, your brain spends less time processing those details and can spend more time retrieving the information you asked for accurately. \n",
    "\n",
    "---\n",
    "\n",
    "## Survey Logic\n",
    "\n",
    "There may be survey questions in which if respondents answer one way, there are certain questions that need to be asked next, but if they answer another way, you'd ideally like to ask them different questions.  This is called *survey logic*, or *skip logic*.  Survey logic is incredibly useful for personalizing the survey experience and ensuring that no one has a longer survey than they need. Here is an example: \n",
    "\n",
    "```text\n",
    "1) Would you be interested in a data scientist training program? \n",
    "-Yes\n",
    "    IF YES: Go to question 2\n",
    "-No\n",
    "    IF NO: Go to question 3\n",
    "\n",
    "2) What would be the ideal length of a data scientist training program? \n",
    "-1 hour\n",
    "-2 hours\n",
    "-3 hours\n",
    "\n",
    "3) Would you be interested in a software engineer training program?\n",
    "-Yes\n",
    "-No\n",
    "```\n",
    "\n",
    "It would be silly to have everyone answer all three questions, because if someone is not interested in a data scientist training program, they probably have no input on how long it should be, and even if they do have input, you probably don't want it, because they are not the target audience.  In the example above, the logic is just roughly blocked out on paper, but if you are doing an online survey, it is easy to convert this into a system in which the participant does not even know they are missing questions, and doesn't have to worry about manually skipping questions.\n",
    "\n",
    "Blocking your survey logic out ahead of time, before going into any software design program, is helpful as a double-check.  You'll spend a lot of time programming survey logic in, and you don't want to make it up on the fly.  \n",
    "\n",
    "---\n",
    "\n",
    "## Thank Them!\n",
    "\n",
    "Once you have finished the survey, it is important to thank your participants! They took time out of their busy day to help you and your company out! Then on the next line, it's always a good idea to reiterate who to contact if they have any questions, now that they have completed the survey.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7b3f800-0a14-4d13-8c32-390965c537ca",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 8 - Survey Review and Testing<a class=\"anchor\" id=\"DS103L4_page_8\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4597e4b4-ce22-448d-843e-eab5ba74a9c4",
   "metadata": {},
   "source": [
    "# Survey Review and Testing\n",
    "\n",
    "Once you think you have a good survey, you'll want to review it extensively to ensure it meets your needs and that it will not harm anyone.  Especially when you are trying to survey a large chunk of the population, there will be no putting the cat back in the bag once it's out! Any mistakes you make will remain. So test, test, test ahead of time!\n",
    "\n",
    "---\n",
    "\n",
    "## Review in Detail\n",
    "\n",
    "The first \"test\" you will want to do is just to review it yourself in excruciating detail.  Next to things like resumes or article publications, a survey is most likely one of the most public things you will undertake, and you want to put your best foot forward. So check, spelling, grammar, design, flow, and any skip logic very carefully yourself. Then hand it to a coworker, friend, or family member for another look.  If that's not an option, read it backwards or out loud, both of which make your brain think about writing in a different way so you can catch typos and other mistakes.\n",
    "\n",
    "---\n",
    "\n",
    "## Have Experts Review\n",
    "\n",
    "Once you've thoroughly reviewed the basics, now it's time to get a subject matter expert (SME)'s opinion on your survey.  They'll be able to review it at a high level to make sure that you are covering all the concepts and that your response options are thorough.  An SME's opinion is invaluable, so do not skip this step!\n",
    "\n",
    "---\n",
    "\n",
    "## Cognitive Interview\n",
    "\n",
    "The remaining testing items on this page go over types of testing you can do with a small number of actual participants that represent the population you are interested in.  The first is *cognitive interviewing*, which is when you give a respondent the survey and ask him or her to think aloud as they answer the survey questions.  The purpose of cognitive interviewing is to understand what the participant is thinking and feeling as he or she goes through the survey.  This will help you identify any issues with inappropriate questions and responses.\n",
    "\n",
    "---\n",
    "\n",
    "## Behavior Coding\n",
    "\n",
    "The next type of test that can be done is *behavior coding*.  When utilizing behavior coding, you will observe a participant as he or she takes your survey to see if there are any behaviors that might indicate trouble.  Does the participant put his or her head down on the desk in despair? Is there lots of frowning and grumbling? What about biting the nails and other nervous tics? If you can record what this behavior is and on which sections of the survey it is occurring, you can identify any problematic areas that might need to be revised. Behavior coding is particularly useful in checking to make sure that your survey does not cause undue distress.\n",
    "\n",
    "---\n",
    "\n",
    "## Response Latency\n",
    "\n",
    "*Response latency* is the process of timing how long a respondent takes to answer each survey question.  The longer the amount of time it takes a participant to answer, the more likely that the question is trouble.  It may indicate that the question is confusing, does not have the right response options, or is emotionally troubling in some way.\n",
    "\n",
    "---\n",
    "\n",
    "## Respondent Debriefing\n",
    "\n",
    "*Respondent debriefing* is probably the most commonly completed survey testing.  It's simple to ask participants to take the survey, and then give feedback on the experience and the questions. In many ways, the testing respondents can serve as an SME as well to review that you have the right questions and response options. You get a lot of valuable information with very little fuss.\n",
    "\n",
    "---\n",
    "\n",
    "## Vignette Analysis\n",
    "\n",
    "All of the above testing requires the participant to take the survey exactly how they are.  However, *vignette analysis* asks survey respondents to answer the survey \"as if\" they were in a hypothetical scenario. So things like, \"respond as if you had a disability,\" or \"respond as if you were pro data integrity,\" etc.  would be asked, so that you could get a feel for how a wide range of participants would respond.  This requires a lot of thought on the part of the participant, however, and usually is only done when you can't get other representatives of the population to test. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36943bcd-7e16-4af1-97d3-c09e81ae5e99",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 9 - Distributing Surveys<a class=\"anchor\" id=\"DS103L4_page_9\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf2cdde-9447-49b5-8936-17d80f46b45e",
   "metadata": {},
   "source": [
    "# Distributing Surveys\n",
    "\n",
    "Now that you have drafted and tested your survey, it's time to distribute it and gather the participants you need to take it! You'll want to think through the best ways to reach your participants with your supervisors and/or clients.  Possible ways to distribute a survey include: \n",
    "\n",
    "* Email\n",
    "* Slack\n",
    "* Host a link on an internal website\n",
    "* Social media sites such as LinkedIn, Facebook, and Twitter\n",
    "* Provide links via QR codes on posters and pamphlets in locations your participants commonly visit\n",
    "* Provide paper copies in locations your participants commonly visit\n",
    "\n",
    "---\n",
    "\n",
    "## Response Rates\n",
    "\n",
    "*Response rate* is the number of people who will click on your survey link.  *Completion rate* is the number of people who will actually finish taking your survey.  Unfortunately, these numbers are not often the same, and you will lose some people taking your survey at the consent page, as they learn more about what will be asked of them, as well as losing some people in the middle of the survey, when it gets too long, too boring, or they otherwise get distracted.  A good survey response rate is about 15%, though of course more is always desired! This means that as you determine who to reach out to, you will need to reach out to many more people than you actually need in order to collect an adequate sample.  For example, if you sent a survey to 100 people, with a survey response rate of 15%, you would only get 15 people.  So, if you did in fact need 100 people, you would need to reach out to 667 people. \n",
    "\n",
    "Plan to keep your survey open at least a month if possible in order to garner as many respondents as possible.\n",
    "\n",
    "---\n",
    "\n",
    "## Remember Your Sampling Lessons\n",
    "\n",
    "How you distribute your survey can have far-reaching implications into the kind of data you collect and how much bias you are introducing to your data.  Although the most rigorous, and thus least biased, sampling strategy is randomly sampling, this is typically impractical when working with most companies, so you'll need to balance rigor with practicality.  Generally think through whether the channels for distribution you plan on using are more likely to pick up certain demographics or types of people.  For instance, if you are trying to entice senior citizens into taking your survey, requiring them to scan a QR code and participate on their phone may not be the best strategy. \n",
    "\n",
    "There is also trouble with a particular type of sampling strategy called *snowball sampling*.  Snowball sampling is when you ask respondents to forward your survey to other people they know.  While it can be helpful when you are trying to survey a small and hard-to-reach population, it should mostly be avoided, because you will bias your data a lot. If those who take the survey pass it on to friends, then your survey is inevitably getting distributed to people that have things in common with the original participant.  Maybe age, gender, race, same health conditions, physical proximity, etc. This means that you are artificially reducing the variation you find within your population and may end up with results wildly different than the actual population.\n",
    "\n",
    "---\n",
    "\n",
    "## Tips to Get Respondents\n",
    "\n",
    "There are numerous research teams devoted to the best way to get survey respondents.  Here are some of the tips and tricks to increase that response rate just a little bit: \n",
    "\n",
    "* **Choose distribution times carefully:** Monday mornings are the best time to distribute; afternoons are the worst. \n",
    "\n",
    "* **Have a trusted authority figure distribute:** Ask a leader, manager, or other trustworthy authority figure to distribute the survey for you.  This must be someone known and respected by your survey respondents.  Further, the person distributing must not only be a reputable figure, but also *look* like a reputable sender.  This means using a company email address, sending things securely, etc. Without this, a lot of people will assume your survey is a scam, which is obviously detrimental. Avoid the spam filter at all costs!\n",
    "\n",
    "* **Have a woman send it:** Especially in male-dominated fields, having a woman's name in the \"from:\" line may increase your response rate. \n",
    "\n",
    "* **Send reminders:** Send reminders at least every two weeks (you may opt for weekly reminders) at peak times (Monday mornings!) from a trusted leader.\n",
    "\n",
    "* **Make survey invitations text based:** Often companies will block images, so ensure that key content is not contained within images and that things will still look good and make sense without rendered images.\n",
    "\n",
    "* **Include a picture of the survey creator:** Even though your images may not be seen, when they are, having an image and some information about the survey creator can boost your response rates a little.  This is especially true (though unfair) if that person is attractive.\n",
    "\n",
    "* **Explain why the survey is important:** Get respondents bought in, and share the lessons you've learned back with them. The more they feel important and needed, the higher your participation will be.\n",
    "\n",
    "* **Include a progress bar:** Including a progress bar will allow participants to know that they are close to being done and may encourage them to finish the survey when they may not have otherwise.\n",
    "\n",
    "---\n",
    "\n",
    "### Offering Rewards\n",
    "\n",
    "To reward or not reward? That 'tis the question! And a much debated question at that! While including a small incentive can increase your response rate, it may also change the motivation of your survey respondents.  They may be more focused on earning money, or on getting whatever the prize is, than answering the questions earnestly and honesty, which at best leads to mistakes on their part and at worse severely biases your survey results. If you do decide to risk using an incentive, it will typically be better to provide a very small reward to everyone who completes the survey, rather than to have people entered into a drawing for a larger prize.  This is particularly true when folks need to provide identifying information at the end in order to be eligible for the raffle; many people will not finish the survey with this point in mind. \n",
    "\n",
    "Whether you choose a reward for everyone or a drawing system, the incentive should not be so large as to be coercive in any way.  People should not feel like they HAVE to take the survey, just because of the reward. This is especially true for monetarily disadvantaged populations.  So, $1 to everyone would be appropriate; $50 to everyone would not be (even if you had the funds). \n",
    "\n",
    "Even if you want to incentivize survey participation, there may be some cases where it is not feasible.  Your company may not have or want to devote the funds that are required, and if you are doing something for the government, it is illegal to spend government money on incentives or other freebies and giveaways.  Knowing your left and right boundaries ahead of time is important! You never want to be in a situation where you have promised an incentive and are not able to provide one.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a0d02d5-04fa-4220-9637-d02c3b32dce5",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 10 - Using Survey Monkey<a class=\"anchor\" id=\"DS103L4_page_10\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1a2c779-b1b9-427f-85e7-cfb274fc19e9",
   "metadata": {},
   "source": [
    "# Using Survey Monkey\n",
    "\n",
    "There are many survey design software packages out there.  If you are working with a company, there's a good bet that you'll utilize a paid software program such as Verint or Qualtrics.  However, there are a lot of free, open-source survey programs out there that you can utilize, and the most common one is *Survey Monkey*. \n",
    "\n",
    "---\n",
    "\n",
    "## Getting Started\n",
    "\n",
    "**[Click here](https://www.surveymonkey.com/)** to go to the SurveyMonkey webpage and get started! You will see a screen like this, and you'll want to click the ```SIGN UP``` button in the upper right hand corner.\n",
    "\n",
    "![The survey monkey website homepage. The title reads, will my product be a success or a flop? There is a button beneath the title that reads, get started. At the top of the screen, a user can log in or sign up.](Media/survey11.png)\n",
    "\n",
    "That will bring you to a screen looking something like below.  Go ahead and sign up - it's free!\n",
    "\n",
    "![A survey monkey screen allowing users to create a free account or log in if the user already has an account. There are fields for a user to create a username and password and to enter their email, first name, and last name. Users can also sign up with Google or Facebook.](Media/survey12.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Create a Survey\n",
    "\n",
    "Once you've set up your account, you will enter a home page that hopefully looks something like this.  Even though things might change some, you are looking for a button that says ```Create Survey```.  Depending on your view, it might be in the upper right corner, in the center of the page, or on the right after discussion of recent surveys. Wherever you find the button, once you click on it, you should see a screen like this one: \n",
    "\n",
    "![A survey monkey screen that allows the user to create a survey. The page says, How do you want to get started? The user can start from scratch or copy an existing survey. There is also a beta option labeled, build it for me. Below this section is a section titled, need some inspiration? In this section a user can search all surveys and templates.](Media/survey13.png)\n",
    "\n",
    "Here you get the option to ```START FROM SCRATCH``` or ```COPY AN EXISTING SURVEY``` along the top.  You can also search for a survey template at the bottom! That's right - for free you can start with some survey questions already set up! Depending on the purpose of the survey, those may work for you, or you may need to create your own.  Copying an existing survey is also really useful when you're doing a routine task, like quarterly progress surveys or something of the ilk.  For learning purposes, you will create your own survey from scratch, so choose that option.  Once you do, you should see this screen:\n",
    "\n",
    "![A survey monkey window that appears when a user creates a survey. It reads, name your survey. The user can provide a name, select demographics, and choose a survey format.](Media/survey14.png)\n",
    "\n",
    "You'll provide your survey with a name, choose the type of survey if you'd like (it doesn't have a lot of bearing on creation), and then choose a survey format. There is some evidence to show that providing only one question at a time can help improve response rate slightly, so unless you have other reasons for showing all the questions on a page, go for that.  Since you wil most likely already have your questions created, you can click that box.  Then click on the big green ```CREATE SURVEY``` button. \n",
    "\n",
    "---\n",
    "\n",
    "## Add Questions\n",
    "\n",
    "Now, you'll be able to copy and paste your entire survey in, if you format it like SurveyMonkey likes - with each question and answer choice on its own line, with enter separating each question.  Forget? Don't worry, SurveyMonkey provides you with everything you need to know: \n",
    "\n",
    "![A survey monkey window that appears when a user adds questions. The user has the ability to see a preview of the survey on the right side of the window, although the preview pane is empty.](Media/survey15.png)\n",
    "\n",
    "Once you've copied and pasted them in, you'll get a preview on the right hand side.  If it's looking right, click the ```ADD QUESTIONS``` button. \n",
    "\n",
    "![A survey monkey window that appears when a user adds questions. On the left are questions. On the right is a preview of the survey.](Media/survey16.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Add Questions from the Question Bank\n",
    "\n",
    "If you want to add new questions from their question bank, that's easy to do. Simply click on the section you need, select the questions, and then click add:\n",
    "\n",
    "![A survey monkey window titled question bank. A user can add question from a question bank to a survey. On the right is a preview.](Media/survey17.png)\n",
    "\n",
    "They will then appear in your question list on the right.\n",
    "\n",
    "---\n",
    "\n",
    "### Add Questions Manually\n",
    "\n",
    "If you want to add questions manually, then you can click on the green ```NEW QUESTION``` button.  This is particularly helpful for questions that are not multiple choice, like matrix questions, as you'll get a whole range of question options here: \n",
    "\n",
    "![A survey monkey screen showing a range of options that a user can select for a question. The options include multiple choice, checkboxes, star rating, dropdown, matrix forward slash rating scale, file upload, net promoter score, image choice, comment box, single textbox, slider, ranking, paying open parentheses stripe close parentheses, matrix of dropdown menus, multiple textboxes, date forward slash time, contact information, text, image, text a forward slash b test, image a forward slash b test.](Media/survey18.png)\n",
    "\n",
    "You can choose anything from a checkbox (choose all that apply) to a text box and more. Of particular note is the option to add Text, where you'll put information about your survey in the consent page and put add additional detail and information about each topic in. Then you'll be able to type in your questions and any answer responses you want to provide. When you're finished, choose ```SAVE``` in the bottom right hand corner. \n",
    "\n",
    "---\n",
    "\n",
    "## Add Pages\n",
    "\n",
    "It may make sense to add pages to your survey, in order to break up your questions logically and consistently.  If you'd like to add a new page, you can simply scroll to the bottom of the survey window and select the ```NEW PAGE``` option. Having new pages is also important, because you can utilize skip logic in SurveyMonkey by page.  So sometimes, you will need to break questions up into a new page based on your logic.\n",
    "\n",
    "![A survey monkey screen showing the new page button, which a user can select to add a new page to a survey in order to break up questions.](Media/survey19.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Add Skip Logic\n",
    "\n",
    "If participants answer a certain way, and you want them to go a particular question afterwards, you can control this in SurveyMonkey by clicking on the yellow ```LOGIC``` button that appears when you hover over the question. It will bring up a menu looking something like this:\n",
    "\n",
    "![A survey monkey screen showing skip logic options during survey creation. Skip logic allows the user of a survey to be sent to a particular question depending on how the user answers a previous question.](Media/survey20.png)\n",
    "\n",
    "From here, you can choose where a participant will go if he or she answers \"ChunkyMonkey\" versus \"Rocky Road\" or any of the other ice cream options. Notice that this goes by page, not by question, so the question you want the participant to hit next will need to be on its own page. Besides listing out pages that you have created, there are also options to send to ```End of survey``` or to ```Disqualification Page```.  These are very useful if you are only targeting a particular demographic or only have permission to survey certain people.  If that happens, you can send them to a disqualification page that you set up, which says something to the effect of: \n",
    "\n",
    "```text\n",
    "You are not eligible to participate in this survey, but thank you for time! If you have any questions, please contact...\n",
    "```\n",
    "\n",
    "You can also have the survey just end, but that is not as polite, and you may lose their participation in future surveys in which they are eligible.\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>A common use for sending someone to a disqualification page is to eliminate those under 18 years of age, as surveying minors often requires specific permissions.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Preview Your Survey\n",
    "\n",
    "Once your survey looks as you'd like it to, click the big green ```NEXT``` button along the top right, which will bring you to a preview page:\n",
    "\n",
    "![A survey monkey screen showing a preview of the first question of a survey, which is a multiple choice question.](Media/survey21.png)\n",
    "\n",
    "Among other things, you'll be able to examine what it looks like on a computer, tablet, and smartphone, and you'll be able to view it in both survey formats to see what you like better. It also estimates the time it will take to complete and the estimated completion rate, which is calculated in part based on survey length.  Beware, as the estimated completion rate may be artificially high!\n",
    "\n",
    "Click ```NEXT``` once more when you are happy with your survey.\n",
    "\n",
    "---\n",
    "\n",
    "## Collect Responses\n",
    "\n",
    "Now you are offered a gazillion ways to collect survey responses: \n",
    "\n",
    "![A survey monkey screen showing options for a survey creator to collect responses to a survey. The options include send by email, get web link, buy responses, post of social media, share in messenger, embed on website, embed in mobile app, add data manually, and kiosk survey.](Media/survey22.png)\n",
    "\n",
    "Most likely, you'll want to go with the option to ```Get Web Link```, although some of the others may also suit your needs. A web link can be placed within an email, added to slack, social media, posted on a website, etc. very easily.  The option you WILL NOT want to choose is to send by email, as it will send a email to folks that doesn't come from a personalized sender and will often get trapped in spam mail, which is a great way to decrease your response rate, as no one will even see your survey link!\n",
    "\n",
    "Once you click on the option to receive a web link, you'll get a page looking something like this one: \n",
    "\n",
    "![A survey monkey screen showing the web link for a survey. The survey creator can customize the link or copy it. The creator can also download a Q R code.](Media/survey23.png)\n",
    "\n",
    "The link is now provided for you, and you can even download a QR code that can be put on print adds and then scanned into a mobile device, which is handy. Your survey is now OPEN, which means that anyone can take it provided they have the link.  It also means that you can't make changes to your survey easily anymore without changing data collection in the middle (never ideal), so for goodness sake, please proof far before you get to this step!\n",
    "\n",
    "There are some options for survey distribution that you'll want to focus on.  Of course, SurveyMonkey puts the paid options on top, which you can ignore, but it's important to look at a few on the bottom:\n",
    "\n",
    "![Survey monkey survey distribution options. Multiple responses, off. Response editing, on, until survey complete. Anonymous responses, off. Instant results, off.](Media/survey24.png)\n",
    "\n",
    "```MULTIPLE OPTIONS``` decides whether participants can take your survey more than once. Typically, you don't want that to happen, so make sure that is ```off```. ```ANONYMOUS RESPONSES``` decides whether your participants will be anonymous.  By default, this is set to off, and so you get the IP address of the respondent.  However, you will typically want this to be ```On```! If you decide to have a non-anonymous survey, you need to disclose it immediately in your consent page.  Those are the two important ones, though you can also edit whether respondents get results and whether or not they can change their answers.  Use your best judgement as to whether you choose those options or not; it will depend on your survey goals.\n",
    "\n",
    "---\n",
    "\n",
    "## Examine Results\n",
    "\n",
    "Lastly, click on the ```ANALYZE RESULTS``` tab to get a copy of your data and be able to look at the results from your survey.  Survey Monkey will automatically display for you some basic information like frequencies, but if you need to do in-depth analysis, it may not suit your needs.  Unfortunately, exporting your data in a useable format like a CSV file is a paid feature in Survey Monkey; you will need to discuss options with your supervisor should you be doing this with a company. \n",
    "\n",
    "![A survey monkey screen titled analyze results. The survey creator can select question summaries, insights and data trends, or individual responses. The main section of the screen reads, your survey has no responses. There are two options beneath, collet responses or buy responses.](Media/survey25.png)\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Wondering how to visualize survey results?</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Check out <a href=\"https://stephanieevergreen.com/rating-data/\"> this blog </a> that goes over how to visualize agree/disagree rating data! </p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b2fd25c-dffa-4217-b2cd-17df1d7dd46c",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 11 - Key Terms<a class=\"anchor\" id=\"DS103L4_page_11\"></a>\n",
    "\n",
    "[Back to Top](#DS103L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58a1fcdd-3889-4a3f-a435-f50e45abc64a",
   "metadata": {},
   "source": [
    "# Key Terms\n",
    "\n",
    "Below is a list and short description of the important keywords learned in this lesson. Please read through and go back and review any concepts you do not fully understand. Great Work!\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Survey</td>\n",
    "        <td>A way to gather information from others through written questions.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Select All Question</td>\n",
    "        <td>Survey question in which participants can select all options that apply.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Scale Question</td>\n",
    "        <td>Asks someone to provide their opinion between anchor choices.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Matrix Question</td>\n",
    "        <td>When multiple scale questions use the same anchors and are combined.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Cognitive Interviewing</td>\n",
    "        <td>Survey testing in which you ask the participant to describe what he or she is thinking as the survey is taken.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Behavior Coding</td>\n",
    "        <td>Observe a participant while he or she takes the survey and note down any unusual behaviors.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Response Latency</td>\n",
    "        <td>Determine how long it takes the respondent to answer each question.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Respondent Debriefing</td>\n",
    "        <td>Ask the participant to give feedback after taking a survey.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Vignette Analysis</td>\n",
    "        <td>Ask the respondent to take the survey in response to a hypothetical scenario.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Response Rate</td>\n",
    "        <td>The number of people who click on your survey link.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Completion Rate</td>\n",
    "        <td>The number of people who finish taking the survey.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Snowball Sampling</td>\n",
    "        <td>A ill-advised sampling strategy in which you ask participants to send the survey to other people they know.</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23e9c74a-516e-4e2c-997e-f2a5123be4f8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}