Merge pull request #472 from fecgov/release/sprint-27

Release/sprint 27

README.md

@@ -1,11 +1,13 @@
 ## About this project
+
 The Federal Election Commission (FEC) is the independent regulatory agency
 charged with administering and enforcing the federal campaign finance law.
 The FEC has jurisdiction over the financing of campaigns for the U.S. House,
 Senate, Presidency and the Vice Presidency.
 
 This project will provide a web application for filling out FEC campaign
 finance information. The project code is distributed across these repositories:
+
 - [fecfile-web-app](https://github.com/fecgov/fecfile-web-app): this is the browser-based front-end developed in Angular
 - [fecfile-web-api](https://github.com/fecgov/fecfile-web-api): RESTful endpoint supporting the front-end
 - [fecfile-validate](https://github.com/fecgov/fecfile-validate): data validation rules and engine
@@ -16,128 +18,151 @@ finance information. The project code is distributed across these repositories:
 ## Set up
 
 ### Prerequisites
+
 Software necessary to run the application locally
 
-* [Docker](https://docs.docker.com/get-docker/)
-* [Docker Compose](https://docs.docker.com/compose/install/)
+- [Docker](https://docs.docker.com/get-docker/)
+- [Docker Compose](https://docs.docker.com/compose/install/)
 
 ### Docker basic usage.
+
 When running docker-compose you will need to be in the root directory of the project. The reason for this is that docker-compose looks for docker-compose.yml to be in the same directory where it's run. You will also need at least 3GB of memory allocated for docker during the build.
 
 ### Run the fecfile web API application
+
 `docker-compose up -d`
 
 You should set the following environment variables in the shell where you are running 'docker-compose up -d'.
 Proper values for the development variables are shown here as an example
+
 ```
 export DATABASE_URL = "postgres://postgres:postgres@db/postgres"
 export FECFILE_TEST_DB_NAME = "postgres"
 export DJANGO_SECRET_KEY = "If_using_test_db_use_secret_key_in_cloud.gov"
 ```
+
 ### Shut down the containers
+
 `docker-compose down`
+
 ### see all running containers
+
 `docker ps`
+
 ### running commands in a running container
-`docker-compose exec <container name> <command>`
 
+`docker-compose exec <container name> <command>`
 
 # Deployment (FEC team only)
 
-*Special Note:* If the fecfile-validate repo was updated, the commit of the update needs to be updated in the requirements.txt file otherwise the CircleCI cache will not roll out the change.
+_Special Note:_ If the fecfile-validate repo was updated, the commit of the update needs to be updated in the requirements.txt file otherwise the CircleCI cache will not roll out the change.
 
 ### Create a feature branch
 
 Using git-flow extensions:
-    ```
-    git flow feature start feature_branch
-    ```
+`   git flow feature start feature_branch
+  `
 
 Without the git-flow extensions:
-    ```
-    git checkout develop
+`   git checkout develop
     git pull
     git checkout -b feature/feature_branch develop
-    ```
+  `
 
-* Developer creates a GitHub PR when ready to merge to `develop` branch
-* Reviewer reviews and merges feature branch into `develop` via GitHub
-* [auto] `develop` is deployed to `dev`
+- Developer creates a GitHub PR when ready to merge to `develop` branch
+- Reviewer reviews and merges feature branch into `develop` via GitHub
+- [auto] `develop` is deployed to `dev`
 
 ### Create a release branch
 
-* Using git-flow extensions:
+- Using git-flow extensions:
+
 ```
 git flow release start sprint-#
 ```
 
-* Without the git-flow extensions:
+- Without the git-flow extensions:
+
 ```
 git checkout develop
 git pull
 git checkout -b release/sprint-# develop
 git push --set-upstream origin release/sprint-#
 ```
-* Developer creates a PR in GitHub to merge release/sprint-# branch into the `main` branch to track if commits pass deployment checks. The actual merge will happen when deploying a release to production.
+
+- Developer creates a PR in GitHub to merge release/sprint-# branch into the `main` branch to track if commits pass deployment checks. The actual merge will happen when deploying a release to production.
 
 ### Create and deploy a hotfix
 
-* Using git-flow extensions:
+- Using git-flow extensions:
+
 ```
 git flow hotfix start my-fix
 # Work happens here
 git flow hotfix finish my-fix
 ```
 
-* Without the git-flow extensions:
+- Without the git-flow extensions:
+
 ```
 git checkout -b hotfix/my-fix main
 # Work happens here
 git push --set-upstream origin hotfix/my-fix
 ```
 
-* Developer creates a hotfix branch, commits changes, and **makes a PR to the `main` and `develop` branches**:
-* Reviewer merges hotfix/my-fix branch into `develop` and `main`
-* [auto] `develop` is deployed to `dev`. Make sure the build passes before deploying to `main`.
-* Developer deploys hotfix/my-fix branch to main using **Deploying a release to production** instructions below
+- Developer creates a hotfix branch, commits changes, and **makes a PR to the `main` and `develop` branches**:
+- Reviewer merges hotfix/my-fix branch into `develop` and `main`
+- [auto] `develop` is deployed to `dev`. Make sure the build passes before deploying to `main`.
+- Developer deploys hotfix/my-fix branch to main using **Deploying a release to production** instructions below
 
 ### Deploying a release to production
-* Reviewer approves PR and merges into `main` (At this point the code is automatically deployed)
-* Check CircleCI for passing pipeline tests
-* If tests pass, continue
-* (If commits were made to release/sprint-#) Developer creates a PR in GitHub to merge release/sprint-# branch into the `develop` branch
-* Reviewer approves PR and merges into `develop`
-* Delete release/sprint-# branch
-* In GitHub, go to `Code -> tags -> releases -> Draft a new release`
-* Publish a new release using tag sprint-#, be sure to Auto-generate release notes
+
+- Reviewer approves PR and merges into `main` (At this point the code is automatically deployed)
+- Check CircleCI for passing pipeline tests
+- If tests pass, continue
+- (If commits were made to release/sprint-#) Developer creates a PR in GitHub to merge release/sprint-# branch into the `develop` branch
+- Reviewer approves PR and merges into `develop`
+- Delete release/sprint-# branch
+- Publish a new release using tag sprint-#, be sure to Auto-generate release notes
+  - On Github, click on "Code" tab, then the "tags" link, then the "Releases" toggle
+  - Click the button "Draft a new release"
+  - Enter the new sprint tag "sprint-XX"
+  - Set Target option to "main"
+  - Set Release title to "sprint-XX"
+  - Click the button "Generate release notes"
+  - Click the "Publish release" button
 
 ## Technical Environment Plan
 
-The fecfile-web-api is our system's backend while the fecfile-web-app is the single-page angular app. The fecfile-web-api is deployed as a cloud.gov application per environment (dev, stage, and prod). Each cloud.gov fecfile-web-api application has at least two instances running.  Similarly, the fecfile-web-app is deployed as a cloud.gov application per environment (dev, stage, and prod).  There are also at least two instances running per cloud.gov fecfile-web-app application.
-
-The following events occur for fecfile-web-api and fecfile-web-app independently of each other:
+The fecfile-web-api is our system's backend while the fecfile-web-app is the single-page angular app. The fecfile-web-api is deployed as a cloud.gov application per environment (dev, stage, and prod). Each cloud.gov fecfile-web-api application has at least two instances running. Similarly, the fecfile-web-app is deployed as a cloud.gov application per environment (dev, stage, and prod). There are also at least two instances running per cloud.gov fecfile-web-app application.
 
-* When a branch is merged into the develop branch, it is deployed to the dev environment on cloud.gov
-	* The Dev environment is used for the bulk of sprint integration and QA testing 
-* When a release is cut (creating a release tag in git), that release is deployed to the stage environment on cloud.gov.
-	* The Stage environment is used for final deployment preparation, integration testing, and final QA testing.  
-* When the release is merged into the main branch, it is deployed to the prod environment on cloud.gov
-	* The Production environment will be used by end users once the application launches.  
+The following events occur for fecfile-web-api and fecfile-web-app independently of each other:
 
+- When a branch is merged into the develop branch, it is deployed to the dev environment on cloud.gov
+  - The Dev environment is used for the bulk of sprint integration and QA testing
+- When a release is cut (creating a release tag in git), that release is deployed to the stage environment on cloud.gov.
+  - The Stage environment is used for final deployment preparation, integration testing, and final QA testing.
+- When the release is merged into the main branch, it is deployed to the prod environment on cloud.gov
+  - The Production environment will be used by end users once the application launches.
 
 ## Additional developer notes
+
 This section covers a few topics we think might help developers after setup.
 
 ### Git Secrets
+
 Set up git secrets to protect oneself from committing sensitive information such as passwords to the repository.
+
 - First install AWS git-secret utility in your PATH so it can be run at the command line: https://github.com/awslabs/git-secrets#installing-git-secrets
 - Once you have git-secrets installed, run the fecfile-web-api/install-git-secrets-hook.sh shell script in the root directory of your cloned fecfile-web-api repo to install the pre-commit hooks.
-NOTE: The pre-commit hook is installed GLOBALLY by default so commits to all cloned repositories on your computer will be scanned for sensitive data. See the comments at the top of the script for local install options.
+  NOTE: The pre-commit hook is installed GLOBALLY by default so commits to all cloned repositories on your computer will be scanned for sensitive data. See the comments at the top of the script for local install options.
 - See git-secrets README for more features: https://github.com/awslabs/git-secrets#readme
 
 ### Commit local code changes to origin daily
+
 As a best practice policy, please commit any feature code changes made during the day to origin each evening before signing off for the day.
 
 ### Google-style inline documentation
+
 The project is using the Google Python Style Guide as the baseline to keep code style consistent across project repositories.
 See here for comment style rules: https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings

django-backend/fecfiler/contacts/serializers.py

@@ -1,6 +1,7 @@
 import logging
 
 from django.db import transaction
+from django.db.models import Q
 from fecfiler.committee_accounts.serializers import CommitteeOwnedSerializer
 from fecfiler.validation import serializers
 from rest_framework.serializers import (
@@ -18,6 +19,7 @@
 class ContactSerializer(
     serializers.FecSchemaValidatorSerializerMixin, CommitteeOwnedSerializer
 ):
+    id = UUIDField(required=False)
     contact_value = dict(
         COM="Committee",
         IND="Individual",
@@ -31,6 +33,20 @@ class ContactSerializer(
     def get_schema_name(self, data):
         return f"Contact_{self.contact_value[data.get('type', None)]}"
 
+    def validate(self, data):
+        matches = (
+            Contact.objects.exclude(id=data.get("id"))
+            .filter(committee_account=data.get("committee_account"))
+            .filter(
+                Q(candidate_id=data.get("candidate_id"), candidate_id__isnull=False)
+                | Q(committee_id=data.get("committee_id"), committee_id__isnull=False)
+            )
+            .count()
+        )
+        if matches != 0:
+            raise ValidationError({"fec_id": ["FEC Ids must be unique"]})
+        return super().validate(data)
+
     class Meta:
         model = Contact
         fields = [
@@ -85,9 +101,9 @@ def update(self, instance, validated_data: dict):
             return super().update(instance, validated_data)
 
     def create_or_update_contact(self, validated_data: dict, contact_key):
-        print(f"AHOY: {validated_data}")
         contact_data = validated_data.pop(contact_key, None)
         contact_id = validated_data.get(contact_key + "_id", None)
+
         if not contact_id:
             if not contact_data:
                 raise ValidationError(

django-backend/fecfiler/contacts/views.py

@@ -4,7 +4,7 @@
 
 import requests
 from django.db.models import CharField, Q, Value, Count
-from django.db.models.functions import Concat, Lower
+from django.db.models.functions import Concat, Lower, Coalesce
 from django.http import HttpResponseBadRequest, JsonResponse
 from fecfiler.committee_accounts.views import CommitteeOwnedViewSet
 from fecfiler.settings import (
@@ -47,12 +47,13 @@ class ContactViewSet(CommitteeOwnedViewSet):
     queryset = (
         Contact.objects.annotate(
             transaction_count=Count("contact_1_transaction_set")
-            + Count("contact_2_transaction_set")
+            + Count("contact_2_transaction_set"),
         )
         .alias(
             sort_name=Concat(
                 "name", "last_name", Value(" "), "first_name", output_field=CharField()
-            )
+            ),
+            sort_fec_id=Coalesce("committee_id", "candidate_id"),
         )
         .all()
     )
@@ -64,6 +65,7 @@ class ContactViewSet(CommitteeOwnedViewSet):
         "type",
         "employer",
         "occupation",
+        "sort_fec_id",
         "id",
     ]
     ordering = ["-created"]
@@ -198,6 +200,19 @@ def organization_lookup(self, request):
 
         return JsonResponse(return_value)
 
+    @action(
+        detail=False,
+        methods=["get"],
+        url_path=r"fec_id_is_unique/(?P<fec_id>[^/.]{0,9})",
+    )
+    def fec_id_is_unique(self, request, fec_id):
+        matches = (
+            self.get_queryset()
+            .filter(Q(candidate_id=fec_id) | Q(committee_id=fec_id))
+            .count()
+        )
+        return Response(matches == 0)
+
     def get_int_param_value(
         self, request, param_name: str, default_param_value: int, max_param_value: int
     ):

django-backend/fecfiler/transactions/schedule_a/tests/test_serializers.py

@@ -9,9 +9,8 @@ class ScheduleATransactionSerializerBaseTestCase(TestCase):
     fixtures = [
         "test_committee_accounts",
         "test_f3x_summaries",
-        "test_contacts",
+        "test_transaction_serializer",
         "test_memo_text",
-        "test_transaction_manager_transactions",
     ]
 
     def setUp(self):
@@ -31,7 +30,7 @@ def setUp(self):
         }
 
         self.updated_contact = {
-            "id": "1578e90c-5348-4afa-9db8-cbeddf9aa701",
+            "id": "00000000-6486-4062-944f-aa0c4cbe4073",
             "type": "IND",
             "last_name": "contact_1",
             "first_name": "Updated",
@@ -91,7 +90,7 @@ def setUp(self):
             "contributor_employer": "boss",
             "report_id": "b6d60d2d-d926-4e89-ad4b-c47d152a66ae",
             "contact_1": self.updated_contact,
-            "contact_1_id": "1578e90c-5348-4afa-9db8-cbeddf9aa701",
+            "contact_1_id": "00000000-6486-4062-944f-aa0c4cbe4073",
             "schema_name": "SchA",
         }
 

django-backend/fecfiler/transactions/schedule_b/managers.py

@@ -22,8 +22,13 @@
     "NON_CONTRIBUTION_ACCOUNT_CREDIT_CARD_PAYMENT_MEMO",
     "NON_CONTRIBUTION_ACCOUNT_PAYMENT_TO_PAYROLL",
     "NON_CONTRIBUTION_ACCOUNT_PAYMENT_TO_PAYROLL_MEMO",
+    "FEDERAL_ELECTION_ACTIVITY_100PCT_PAYMENT",
     "FEDERAL_ELECTION_ACTIVITY_CREDIT_CARD_PAYMENT",
+    "FEDERAL_ELECTION_ACTIVITY_CREDIT_CARD_PAYMENT_MEMO",
+    "FEDERAL_ELECTION_ACTIVITY_STAFF_REIMBURSEMENT",
+    "FEDERAL_ELECTION_ACTIVITY_STAFF_REIMBURSEMENT_MEMO",
     "FEDERAL_ELECTION_ACTIVITY_PAYMENT_TO_PAYROLL",
+    "FEDERAL_ELECTION_ACTIVITY_PAYMENT_TO_PAYROLL_MEMO",
     "RECOUNT_ACCOUNT_DISBURSEMENT",
     "NATIONAL_PARTY_RECOUNT_ACCOUNT_DISBURSEMENT",
     "NATIONAL_PARTY_HEADQUARTERS_ACCOUNT_DISBURSEMENT",

django-backend/fecfiler/transactions/schedule_b/tests/test_serializers.py

@@ -13,13 +13,12 @@ class ScheduleBTransactionSerializerBaseTestCase(TestCase):
     fixtures = [
         "test_committee_accounts",
         "test_f3x_summaries",
-        "test_contacts",
         "test_memo_text",
+        "test_transaction_serializer",
     ]
 
     def setUp(self):
         self.new_contact = {
-            "id": "9bb5c8b2-31f3-488f-84e1-a63b0133a000",
             "type": "IND",
             "last_name": "contact_1",
             "first_name": "New",
@@ -34,13 +33,13 @@ def setUp(self):
         }
 
         self.updated_contact = {
-            "id": "a5061946-93ef-47f4-82f6-f1782c333d70",
+            "id": "00000000-6486-4062-944f-aa0c4cbe4073",
             "type": "IND",
             "last_name": "Lastname",
             "first_name": "Firstname",
             "street_1": "Street",
             "city": "City",
-            "state": "State",
+            "state": "MD",
             "zip": "12345678",
             "country": "Country",
             "created": "2022-02-09T00:00:00.000Z",
@@ -94,8 +93,8 @@ def setUp(self):
             "payee_occupation": "professional",
             "payee_employer": "boss",
             "report_id": "b6d60d2d-d926-4e89-ad4b-c47d152a66ae",
-            "contact_1": self.new_contact,
-            "contact_1_id": "a5061946-93ef-47f4-82f6-f1782c333d70",
+            "contact_1": self.updated_contact,
+            "contact_1_id": "00000000-6486-4062-944f-aa0c4cbe4073",
             "schema_name": "SchB",
         }