Merge branch 'main' into 1085-fix-exporting-with-media

praekeltfoundation · Feb 19, 2024 · b724b26 · b724b26
2 parents 3209906 + a07f1ac
commit b724b26
Show file tree

Hide file tree

Showing 27 changed files with 333 additions and 80 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,19 +8,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 <!--
 ## Unreleased
 
+### Fixed
+
+### Added
+
+### Changed
+
+-->
+## v1.1.0
 ### Fixed
 - Autofill for empty slug
+- Fixed Web content preview in CMS
+- Fixed related pages export
+- Fixed API tests
+- Fixed Redis caching issues
 
 ### Added
+- Added support for SMS content
+- Added support for USSD content
+- Added support for a Footer in WhatsApp content
+- Added support for List Messages in WhatsApp content
+- Added error handling on WhatsApp template submission errors, and adds issue to sentry
+- Added validation of variables used in WhatsApp template submission
+- Added WhatsApp title check
+- Added typing information for migration test
+- Added CI checks for pending migrations
+
 
 ### Changed
 - Moved slug uniqueness validation to the model validation/clean
 - Empty slugs will auto-generate a unique slug, but if a duplicate slug is specified the user will get a validation error instead of their chosen slug getting overwritten with a unique one.
 - Slug uniqueness is per-locale
-- Test speedups. Tests now run in parallel by default, and there's a separate contentrepo/settings/test.py for test-specific settings.
-- Tests no longer run in parallel by default, because the output is a little less clear and the speedup is negligible on some systems.
+- Test speedups, and there's a separate contentrepo/settings/test.py for test-specific settings.
+- WhatsApp example values no longer show on the default API view, and forms part of the fields returned when the parameter `whatsapp=true` is added to the api call
+- Improved testing of Importer
+- Made text fields non-nullable
+- Converted API tests to pytest and added docstrings
+- Updated release process in Readme.
+
+### Deprecated
+ - Removed the old importer
+ - Removed the `Authorization` and `Authentication` WhatsApp template category
 
--->
 
 ## v1.1.0-dev.5
 ### Fixed
@@ -55,6 +84,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Refactored ContentPage import. Translation keys are now added as translation keys instead of as tags
 - Improved testing for import and export
 
+
 ### Deprecated
 - next_prompt on WhatsApp messages is deprecated, use buttons instead. Will be removed on the next major release
 

diff --git a/README.md b/README.md
@@ -28,26 +28,28 @@ This content repository allows easy content management via a headless CMS, using
 
 
 ## Releases
-The current LTS release is: 1.0
-
 [Semantic versioning](https://semver.org/) is in use in this project.
-1. Feature releases
-    - Feature releases are a MAJOR.MINOR combination, eg. `1.2`
-    - Feature releases occur once every 3 months, but only if required. eg. If there were only patch releases in the last 3 months, no feature release will occur.
-    - The latest feature release will receive any security or bug fixes as patch releases
-    - Each feature release will have a separate git branch
-1. Patch releases
-    - Patch releases are used to fix bugs and security issues
-    - They are released as soon as they are ready
-1. LTS (Long-Term Support) Releases
-    - Every fourth release is an LTS release
-    - LTS releases are listed in the documentation
-    - LTS releases receive security patches
-    - LTS releases are supported for 5 feature releases, allowing for 15 months of support with a 3 month switchover time to the next LTS
+
 1. Development releases
-    - Development have the `-dev.N` suffix
-    - They are used to test new code before an official release is made
+    - Development versions have the `-dev.N` suffix
+    - They are built off of the `main` git branch
+    - They are created with git tags, eg. `v1.2.0-dev.2`
+    - Development releases are created before releases, eg. `v1.2.0-dev.0` gets created before `v1.2.0`
+1. Releases
+    - Releases have a semantic version number, eg. `1.2.0`
+    - Releases are created when the changes have passed QA
     - They are built off of the `main` git branch
+    - They are created with git tags, eg. `v1.2.0`
+
+### How to create a release
+
+1. Check that `CHANGELOG.md` is up to date.
+1. Update the version number in `pyproject.toml` to the version you want to release, eg. change `1.2.0-dev.0` to `1.2.0`
+1. Either create a pull request with these changes, and have it reviewed, or post a diff in Slack and have it reviewed there. Then merge, or commit and push, the changes.
+1. Tag these changes with the desired release, eg. `git tag v1.2.0`, and push the tag, eg. `git push --tags`
+1. Update the version number in `pyproject.toml` to the next version number, eg. `1.2.1-dev.0`.
+1. Commit and push that change.
+
 
 ## Setting up locally
 

diff --git a/contentrepo/settings/test.py b/contentrepo/settings/test.py
@@ -9,3 +9,5 @@
 FB_BUSINESS_ID = "27121231234"
 
 WHATSAPP_CREATE_TEMPLATES = False
+
+STATICFILES_STORAGE = "django.contrib.staticfiles.storage.StaticFilesStorage"
diff --git a/home/export_content_pages.py b/home/export_content_pages.py
@@ -1,5 +1,6 @@
 import copy
 import csv
+import io
 from dataclasses import asdict, astuple, dataclass, fields
 from itertools import zip_longest
 from json import dumps
@@ -77,6 +78,7 @@ class ExportRow:
     doc_link: str = ""
     media_link: str = ""
     related_pages: str = ""
+    footer: str = ""
 
     @classmethod
     def headings(cls) -> list[str]:
@@ -133,8 +135,14 @@ def add_message_fields(self, msg_blocks: MsgBlocks) -> None:
                 self.buttons = self.serialise_buttons(whatsapp.value["buttons"])
             if "example_values" in whatsapp.value:
                 self.example_values = ", ".join(whatsapp.value["example_values"])
+            if "footer" in whatsapp.value:
+                self.footer = whatsapp.value["footer"]
             if "list_items" in whatsapp.value:
-                self.list_items = ", ".join(whatsapp.value["list_items"])
+                output = io.StringIO()
+                writer = csv.writer(output)
+                writer.writerow(whatsapp.value["list_items"])
+                self.list_items = output.getvalue().strip()
+                output.close()
 
     @staticmethod
     def serialise_buttons(buttons: blocks.StreamValue.StreamChild) -> str:
@@ -344,6 +352,7 @@ def _set_xlsx_styles(wb: Workbook, sheet: Worksheet) -> None:
         "doc_link": 118,
         "media_link": 118,
         "related": 118,
+        "footer": 118,
     }
 
     for index, column_width in enumerate(column_widths_in_pts.values(), 2):

diff --git a/home/import_content_pages.py b/home/import_content_pages.py
@@ -258,6 +258,13 @@ def create_shadow_content_page_from_row(
             parent=row.parent,
             related_pages=row.related_pages,
         )
+
+        if len(row.footer) > 60:
+            raise ImportException(f"footer too long: {row.footer}", row.page_id)
+
+        if len(row.list_items) > 24:
+            raise ImportException(f"list_items too long: {row.list_items}", row.page_id)
+
         self.shadow_pages[(row.slug, locale)] = page
 
         self.add_message_to_shadow_content_page_from_row(row, locale)
@@ -326,6 +333,7 @@ def add_message_to_shadow_content_page_from_row(
                     next_prompt=row.next_prompt,
                     example_values=row.example_values,
                     buttons=buttons,
+                    footer=row.footer,
                     list_items=row.list_items,
                 )
             )
@@ -530,6 +538,7 @@ class ShadowWhatsappBlock:
     example_values: list[str] = field(default_factory=list)
     variation_messages: list["ShadowVariationBlock"] = field(default_factory=list)
     list_items: list[str] = field(default_factory=list)
+    footer: str = ""
 
     @property
     def wagtail_format(
@@ -542,6 +551,7 @@ def wagtail_format(
             "buttons": self.buttons,
             "variation_messages": [m.wagtail_format for m in self.variation_messages],
             "list_items": self.list_items,
+            "footer": self.footer,
         }
 
 
@@ -631,6 +641,7 @@ class ContentRow:
     doc_link: str = ""
     media_link: str = ""
     related_pages: list[str] = field(default_factory=list)
+    footer: str = ""
 
     @classmethod
     def from_flat(cls, row: dict[str, str]) -> "ContentRow":
@@ -651,6 +662,7 @@ def from_flat(cls, row: dict[str, str]) -> "ContentRow":
             example_values=deserialise_list(row.pop("example_values", "")),
             buttons=json.loads(row.pop("buttons", "")) if row.get("buttons") else [],
             list_items=deserialise_list(row.pop("list_items", "")),
+            footer=row.pop("footer") if row.get("footer") else "",
             **row,
         )
 
@@ -714,4 +726,6 @@ def deserialise_dict(value: str) -> dict[str, str]:
 def deserialise_list(value: str) -> list[str]:
     if not value:
         return []
-    return [item.strip() for item in value.strip().split(",")]
+
+    items = list(csv.reader([value]))[0]
+    return [item.strip() for item in items]
diff --git a/home/migrations/0048_alter_contentpage_whatsapp_body.py b/home/migrations/0048_alter_contentpage_whatsapp_body.py
@@ -1,4 +1,4 @@
-# Generated by Django 4.1.13 on 2024-01-30 15:25
+# Generated by Django 4.1.10 on 2024-02-01 13:02
 
 import django.core.validators
 import wagtail.blocks
@@ -190,6 +190,18 @@ class Migration(migrations.Migration):
                                         ),
                                     ),
                                 ),
+                                (
+                                    "footer",
+                                    wagtail.blocks.CharBlock(
+                                        help_text="Footer cannot exceed 60 characters.",
+                                        required=False,
+                                        validators=(
+                                            django.core.validators.MaxLengthValidator(
+                                                60
+                                            ),
+                                        ),
+                                    ),
+                                ),
                             ],
                             help_text="Each message will be sent with the text and media",
                         ),
@@ -201,3 +213,4 @@ class Migration(migrations.Migration):
             ),
         ),
     ]
+
diff --git a/home/models.py b/home/models.py
@@ -1,3 +1,4 @@
+import logging
 import re
 
 from django.conf import settings
@@ -40,7 +41,6 @@
     TabbedInterface,
     TitleFieldPanel,
 )
-import logging
 
 logger = logging.getLogger(__name__)
 
@@ -265,6 +265,12 @@ class WhatsappBlock(blocks.StructBlock):
         validators=(MaxLengthValidator(24)),
     )
 
+    footer = blocks.CharBlock(
+        help_text="Footer cannot exceed 60 characters.",
+        required=False,
+        validators=(MaxLengthValidator(60),),
+    )
+
     class Meta:
         icon = "user"
         form_classname = "whatsapp-message-block struct-block"

diff --git a/home/templates/wagtail_content_import/picker_buttons_base.html b/home/templates/wagtail_content_import/picker_buttons_base.html
@@ -0,0 +1,35 @@
+{% if default_picker %}
+    <li class="actions import-container content-import">
+        <div class="dropdown dropup dropdown-button match-width">
+            <div class="content-import button button-longrunning dropdown-toggle">
+                <i class="icon icon-folder-open-inverse content-import docs-blue"></i>
+                Import web from doc
+            </div>
+            <div class="content-import dropdown-toggle icon icon-arrow-up"></div>
+            <ul>
+                <li>
+                    <a href="#"
+                    data-content-import-picker="{{ default_picker.name }}"
+                    data-import-page-url="{% block default_import_page_url %}{% endblock %}"
+                    {% block default_extra_attrs %}{% endblock %}
+                    class="content-import button button-longrunning">
+                        <i class="icon {{ default_picker.icon }} content-import docs-blue"></i>
+                        Import from {{ default_picker.verbose_name }}
+                    </a>
+                </li>
+                {% for picker in picker_options %}
+                    <li>
+                        <a href="#"
+                           data-content-import-picker="{{ picker.name }}"
+                           data-import-page-url="{% block import_page_url %}{% endblock %}"
+                           {% block extra_attrs %}{% endblock %}
+                           class="content-import button button-longrunning">
+                            <i class="icon {{ picker.icon }} content-import docs-blue"></i>
+                            Import from {{ picker.verbose_name }}
+                        </a>
+                    </li>
+                {% endfor %}
+            </ul>
+        </div>
+    </li>
+{% endif %}
diff --git a/home/tests/content2.csv b/home/tests/content2.csv
@@ -1,5 +1,5 @@
-structure,message,page_id,slug,parent,web_title,web_subtitle,web_body,whatsapp_title,whatsapp_body,variation_title,variation_body,list_items,sms_title,sms_body,ussd_title,ussd_body,messenger_title,messenger_body,viber_title,viber_body,translation_tag,tags,quick_replies,triggers,locale,next_prompt,buttons,image_link,doc_link,media_link,related_pages,example_values
-Menu 1,0,,main-menu,,Main Menu,,,,,,,,,,,,,,,,a0b85075-d01b-46bf-8997-8591e87ba171,,,,English,,,,,,,[]
+structure,message,page_id,slug,parent,web_title,web_subtitle,web_body,whatsapp_title,whatsapp_body,variation_title,variation_body,list_items,sms_title,sms_body,ussd_title,ussd_body,messenger_title,messenger_body,viber_title,viber_body,translation_tag,tags,quick_replies,triggers,locale,next_prompt,buttons,image_link,doc_link,media_link,related_pages,example_values,footer
+Menu 1,0,,main-menu,,Main Menu,,,,,,,,,,,,,,,,a0b85075-d01b-46bf-8997-8591e87ba171,,,,English,,,,,,,[],
 Sub 1.1,1,,main-menu-first-time-user,Main Menu,main menu first time user,,,main menu first time user," *Welcome to HealthAlert* 🌍
 
 This is a messaging service created by the _*World Health Organization*_ *(WHO)* that provides information on COVID-19 as well as emergency resources for disease outbreaks, natural, and man-made disasters.
@@ -12,12 +12,12 @@ This is a messaging service created by the World Health Organization (WHO) that
 
 You can return to this main menu at any time by replying 🏠
 
-Choose what you'd like to know more about by tapping a button below ⬇️",,,5892bccd-8025-419d-9a8e-a6a37b755dbf,menu,"Self-help🌬️, Settings⚙️, Health Info🏥",Main menu🏠,English,,[],,,,,[]
+Choose what you'd like to know more about by tapping a button below ⬇️",,,5892bccd-8025-419d-9a8e-a6a37b755dbf,menu,"Self-help🌬️, Settings⚙️, Health Info🏥",Main menu🏠,English,,[],,,,,[],
 Sub 1.1.1,1,,health-info,main menu first time user,health info,,,health info,"*Health information* 🏥
 
 Get information and advice from WHO by tapping on a button below ⬇️",,,,Health Info SMS Title,*Health Info SMS Body*,Health Info USSD Title,*Health Info USSD Body*,health info,"*Health information* 🏥
 
-Get information and advice from WHO by tapping on a button below ⬇️",,,c9d6309e-173f-4c1d-bbaf-440b1fd4415f,health_info,,,English,,[],,,,,[]
+Get information and advice from WHO by tapping on a button below ⬇️",,,c9d6309e-173f-4c1d-bbaf-440b1fd4415f,health_info,,,English,,[],,,,,[],
 Sub 1.1.2,1,,self-help,main menu first time user,self-help,,,self-help,"*Self-help programs* 🌬️
 
 Reply with a number to take part in a *free* self-help program created by WHO.
@@ -32,4 +32,4 @@ Reply with a number to take part in a *free* self-help program created by WHO.
 1. Quit tobacco 🚭
 _Stop smoking with the help of a guided, 42-day program._
 2. Manage your stress 🧘🏽‍♀️
-_Learn how to cope with stress and improve your wellbeing._",,,3e5d77f7-4d34-430d-aad7-d9ca01f79732,self_help,,,English,,[],,,,,[]
+_Learn how to cope with stress and improve your wellbeing._",,,3e5d77f7-4d34-430d-aad7-d9ca01f79732,self_help,,,English,,[],,,,,[],
diff --git a/home/tests/exported_content_20230911-variations-linked-page.csv b/home/tests/exported_content_20230911-variations-linked-page.csv
@@ -1,12 +1,12 @@
-structure,message,page_id,slug,parent,web_title,web_subtitle,web_body,whatsapp_title,whatsapp_body,whatsapp_template_name,variation_title,variation_body,list_items,sms_title,sms_body,ussd_title,ussd_body,messenger_title,messenger_body,viber_title,viber_body,translation_tag,tags,quick_replies,triggers,locale,next_prompt,buttons,image_link,doc_link,media_link,related_pages,example_values
-Menu 1,0,5,appointment-reminders,,Appointment reminders,,,,,,,,,,,,,,,,,8cff04aa-cf08-4120-88b4-e2269b7d5d80,,,,English,,,,,,,
-Menu 2,0,6,stage-based-messages,,Stage-based messages,,,,,,,,,,,,,,,,,5f2d9e63-f047-41ab-921a-c5ca7c04d643,,,,English,,,,,,,
-Menu 3,0,7,health-info-messages,,Health info messages,,,,,,,,,,,,,,,,,3db069a4-7112-4e66-a9a2-f35c6c18055a,,,,English,,,,,,,
-Menu 4,0,17,whatsapp-template-testing,,whatsapp template testing,,,,,,,,,,,,,,,,,5f7221f4-146a-48c2-b2e3-c8491aaead9d,,,,English,,,,,,,
-Menu 5,0,164,import-export,,Import Export,,,,,,,,,,,,,,,,,497bdc1f-43fc-4925-80a1-e68cb942faa4,,,,English,,,,,,,
-Sub 5.1,1,165,cp-import-export,Import Export,CP-Import/export,,,WA import export data,Message 1 contains an image,,,,,,,,,,,,,8ac50daf-de21-4d05-b697-6d983b7ed3d5,"Tag2, Tag1",Quick reply1,"Trigger2, Trigger1",English,,[],/admin/images/usage/4/,,,ma_qa_temp,
-,1,165,cp-import-export,,,,,,,,gender: male,Variation message one for Gender Male,,,,,,,,,,,,,,,,,,,,,
-,2,165,cp-import-export,,,,,,"Message2 has a document attached, lets add some variable placeholders as well {{0}}",,,,,,,,,,,,,,,,,,,[],,/admin/documents/usage/1/,,,
-,2,165,cp-import-export,,,,,,,,gender: empty,Variation message one for Gender Rather not say,,,,,,,,,,,,,,,,,,,,,
-,3,165,cp-import-export,,,,,,Message 3 with no variation but has an audio clip,,,,,,,,,,,,,,,,,,,[],,,/admin/media/usage/1/,,
-Sub 5.2,1,166,ma_qa_temp,Import Export,MA QA Temp,,,,,,,,,,,,,,,,,e9793b5f-f8c7-46c5-8a5e-bd9b8f00fee9,,,,English,,,,,,,
+structure,message,page_id,slug,parent,web_title,web_subtitle,web_body,whatsapp_title,whatsapp_body,whatsapp_template_name,variation_title,variation_body,list_items,sms_title,sms_body,ussd_title,ussd_body,messenger_title,messenger_body,viber_title,viber_body,translation_tag,tags,quick_replies,triggers,locale,next_prompt,buttons,image_link,doc_link,media_link,related_pages,example_values,footer
+Menu 1,0,5,appointment-reminders,,Appointment reminders,,,,,,,,,,,,,,,,,8cff04aa-cf08-4120-88b4-e2269b7d5d80,,,,English,,,,,,,,
+Menu 2,0,6,stage-based-messages,,Stage-based messages,,,,,,,,,,,,,,,,,5f2d9e63-f047-41ab-921a-c5ca7c04d643,,,,English,,,,,,,,
+Menu 3,0,7,health-info-messages,,Health info messages,,,,,,,,,,,,,,,,,3db069a4-7112-4e66-a9a2-f35c6c18055a,,,,English,,,,,,,,
+Menu 4,0,17,whatsapp-template-testing,,whatsapp template testing,,,,,,,,,,,,,,,,,5f7221f4-146a-48c2-b2e3-c8491aaead9d,,,,English,,,,,,,,
+Menu 5,0,164,import-export,,Import Export,,,,,,,,,,,,,,,,,497bdc1f-43fc-4925-80a1-e68cb942faa4,,,,English,,,,,,,,
+Sub 5.1,1,165,cp-import-export,Import Export,CP-Import/export,,,WA import export data,Message 1 contains an image,,,,,,,,,,,,,8ac50daf-de21-4d05-b697-6d983b7ed3d5,"Tag2, Tag1",Quick reply1,"Trigger2, Trigger1",English,,[],/admin/images/usage/4/,,,ma_qa_temp,,
+,1,165,cp-import-export,,,,,,,,gender: male,Variation message one for Gender Male,,,,,,,,,,,,,,,,,,,,,,
+,2,165,cp-import-export,,,,,,"Message2 has a document attached, lets add some variable placeholders as well {{0}}",,,,,,,,,,,,,,,,,,,[],,/admin/documents/usage/1/,,,,
+,2,165,cp-import-export,,,,,,,,gender: empty,Variation message one for Gender Rather not say,,,,,,,,,,,,,,,,,,,,,,
+,3,165,cp-import-export,,,,,,Message 3 with no variation but has an audio clip,,,,,,,,,,,,,,,,,,,[],,,/admin/media/usage/1/,,,
+Sub 5.2,1,166,ma_qa_temp,Import Export,MA QA Temp,,,,,,,,,,,,,,,,,e9793b5f-f8c7-46c5-8a5e-bd9b8f00fee9,,,,English,,,,,,,,
Original file line number	Diff line number	Diff line change
Expand Up		@@ -9,3 +9,5 @@
		FB_BUSINESS_ID = "27121231234"

		WHATSAPP_CREATE_TEMPLATES = False

		STATICFILES_STORAGE = "django.contrib.staticfiles.storage.StaticFilesStorage"