Merge branch 'development' of github.com:hotosm/fmtm into development

CHANGELOG.md

@@ -1,5 +1,83 @@
 # Changelog
 
+## 2025.1.1 (2025-02-26)
+
+### Feat
+
+- separate out primary mapping geometry from new feature geometry (during proj create) (#2225)
+- **frontend**: start user roles integration (#2207)
+- **frontend**: task comments for individual features( #2218)
+- **backend**: add task scheduler to backend services (cron jobs) (#2147)
+- finalise S3 submission photos on backend + frontend (#2211)
+- **backend**: replace custom ODK submission media upload with official external storage (S3) (#1894)
+- **backend**: replace nominatim usage with pg-nearest-city reverse-geocode (#2199)
+- **backend**: extract additional geoms into submission geojson featcol
+- **frontend**: design modifications, project details, home page, header, buttons (#2148)
+- **frontend**: show geometry label popups on submission details page (#2187)
+- **frontend**: download split geojson during proj create (#2186)
+- **frontend**: download filtered submissions, with option for geojson too (#2183)
+- **backend**: endpoints to download form XML and create new form submissions (#2178)
+- create script for generating project stats via API
+- **frontend**: render points on both frontends, with colour determined by status (#2174)
+- **submission**: add date range filter for geojson download (#2176)
+- **mapper**: add i18n (internationalisation) frontend translations via paraglide js (#2155)
+- **tasks**: add task to unlock tasks locked for over 3 days and reset entities after an hour (#1984)
+- **backend**: return entity uuid on create (#2151)
+- **backend**: add endpoint to create an entity in existing project for new geometry (#2145)
+- **frontend**: update user roles list during project creation (#2135)
+- **frontend**: add table component and manage users page (#2133)
+- **backend**: add central-webhook service for triggering entity status updates in FMTM database (#2130)
+- **backend**: migration to add dataset property to old projects (#2126)
+- **users**: add pagination and search functionality (#2124)
+- **backend**: add endpoint to change global user roles (#2117)
+- **frontend**: assign PM during project creation & refactor radix components (#2115)
+- **backend**: warn users and delete user accounts after period of inactivity  (#2088)
+
+### Fix
+
+- **dialog-entities-actions**: update submission length check condition
+- **backend**: set new_geom_type default to Polygon as temp fix for #2164
+- **taskSelectionPopup**: update task popup task state label
+- **submissionDetails**: display submission images on the submission field itself (#2216)
+- add project create feature warning (10,000) and limit (30,000) (#2215)
+- **frontend**: inconsistent task history names and styling (#2217)
+- **backend**: update role to project admin to fetch user list (#2203)
+- **frontend**: correctly include project name in geojson submission download file
+- **frontend**: use fmtm-cursor-pointer for default user icon
+- **frontend**: move new geom confirm dialog to center screen
+- **extractGeojsonFromObject**: differentiate and visualize Polygon & LineString seperately (#2196)
+- **frontend**: task lock logic, allow unlock by admins (#2188)
+- **+page**: confirmation dialog close before redirection to ODK (#2185)
+- **frontend**: add error message if project editing fails (permissions)
+- **frontend**: add hot-tracking to react type declarations
+- **backend**: TMS issues by using go-tilepacks and go-pmtiles (#2162)
+- **backend**: support big int osm id fgb creation (#2160)
+- **backend**: clear tiles from disk after generation (#2158)
+- updated usage of shapestream requiring 'params' field
+- **projectDetails**: update set odk credential logic (#2149)
+- **frontend**: custom odk creds validation, relate to PR #2142
+- **frontend**: allow users to select default odk credentials (#2142)
+- **getTaskStatusStyle**: fix locked_for_mapping task state color
+- **backend**: getting secret value from CENTRAL_WEBHOOK_API_KEY config var
+- **frontend**: logic allowing for custom odk server creds during proj create (#2129)
+- **frontend**: validated entity state visualization on map (#2122)
+- **mapper**: flaky new feature polygon draw (#2118)
+
+### Refactor
+
+- **frontend**: replace old button with new button components (#2212)
+- **frontend**: update link to custom ODK Collect --> v2024.3.5 (entity refresh on load)
+- automatically determine geom type from javarosa geom structure
+- remove sign out button from sidebar drawer
+- replace two submission download endpoints with one
+- tweak odk tunnel testing config button
+- **frontend**: remove task_filter field from frontend intent url
+- **frontend**: simplify SetSnackBar action with defaults (#2192)
+- use @types/react for custom module declarations
+- ensure translations match
+- move unused root level 'images' to prototyping docs section
+- **submissionDetails**: fix bracket, don't show new_feature geometry object (#2121)
+
 ## 2025.1.0 (2025-01-24)
 
 ### Feat

README.md

@@ -123,11 +123,11 @@ Alternatively see the [docs](https://docs.fmtm.dev) for various deployment guide
 |✅| 📱 features turn green once mapped |
 |✅| 📱 better support for mapping **new** points, lines, polygons |
 |✅| 📱 navigation and capability for routing to map features |
+|✅| 🖥️ organization creation and management |
 |⚙️| 📱 integrate ODK Web Forms (to avoid switching apps) |
 |⚙️| 🖥️ multiple approaches to task splitting algorithm |
 |⚙️| 🖥️ user role management per project |
 | | 📱 fully offline field mapping |
-| | 🖥️ organization creation and management |
 | | 🖥️ simplify project creation with basic / advanced workflows |
 | | 🖥️ improvements to the validation criteria and workflow |
 | | 🖥️ export (+merge) the final data to OpenStreetMap |

chart/Chart.yaml

@@ -3,7 +3,7 @@ type: application
 name: fmtm
 description: Field Mapping Tasking Manager - coordinated field mapping.
 version: "0.1.0"
-appVersion: "2025.1.0"
+appVersion: "2025.1.1"
 maintainers:
 - email: [email protected]
   name: Sam Woodcock

deploy/compose.main.yaml

@@ -104,13 +104,25 @@ services:
       file: compose.staging.yaml
       service: scheduler
 
-  backup:
-    extends:
-      file: compose.staging.yaml
-      service: backup
+  backups:
+    image: "ghcr.io/hotosm/fmtm/backend:main"
     depends_on:
       fmtm-db:
         condition: service_healthy
+      s3:
+        condition: service_healthy
+    env_file:
+      - .env
+    networks:
+      - fmtm-net
+    entrypoint: ["/backup-entrypoint.sh"]
+    restart: "on-failure:2"
+    healthcheck:
+      test: pg_isready -U ${FMTM_DB_USER} -d ${FMTM_DB_NAME}
+      start_period: 5s
+      interval: 10s
+      timeout: 5s
+      retries: 3
 
   certbot:
     image: "ghcr.io/hotosm/fmtm/proxy:certs-init-main"

docs/manuals/project-managers.md

@@ -138,9 +138,8 @@ Alternatively, request the creation of a new organisation for your team:
 
     ![image](https://github.com/user-attachments/assets/64aeda34-c682-4fdc-8c2f-1fd83e29c61f)
 
-13. The step 3 is to choose the form category of the project. Meaning if you want
-    to survey each household or healthcare or educational institutes.
-    You can upload the custom XLS form by clicking on the checkbox.
+13. Upload your XLSForm. Here you download pre-defined forms from FMTM.
+    Some are specifically designed to work with OpenStreetMap.
     Click on "Next" to proceed.
 
     ![image](https://github.com/user-attachments/assets/cdf1e050-42ec-4149-bf97-0d841bc5117f)

docs/manuals/xlsform-design.md

@@ -157,10 +157,10 @@ overview of the injected fields and their purposes:
 4. If no feature is selected, the user would be prompted to take a GPS coordinate
    of new feature.
    Note: One of these two options must be filled up to proceed.
-5. We also dedicate few rows for calculating form category, osm ID,
-   Task Id and mapping status used on FMTM.
+5. We also dedicate few rows for calculating OSM ID,
+   Task ID and mapping status used on FMTM.
 6. We then ask mappers to answer if the feature exist in reality?
-   If yes, the custom form uploaded by user is proceeded.
+   If yes, user proceeds with form submission.
 7. If no, the user is prompted to capture an image (if available) and the form
    is terminated with a message:
    "You cannot proceed with data acquisition if the building does not exist."

src/backend/Dockerfile

@@ -191,7 +191,9 @@ EOT
 CMD ["python", "-Xfrozen_modules=off", "-m", "debugpy", \
     "--listen", "0.0.0.0:5678", "-m", "uvicorn", "app.main:api", \
     "--host", "0.0.0.0", "--port", "8000", "--workers", "1", \
-    "--reload", "--log-level", "critical", "--no-access-log"]
+    "--reload", "--reload-dir", "app", "--reload-dir", "tests", \
+    "--reload-dir", "scheduler", "--reload-dir", "stats", "--log-level", \
+    "critical", "--no-access-log"]
 
 
 # Used during CI workflows (as root), with docs/test dependencies pre-installed

src/backend/app/__version__.py

@@ -1 +1 @@
-__version__ = "2025.1.0"
+__version__ = "2025.1.1"

src/backend/app/auth/roles.py

@@ -289,11 +289,14 @@ async def wrap_check_access(
 
 
 async def project_manager(
-    project: Annotated[DbProject, Depends(get_project)],
+    project_id: int,
     db: Annotated[Connection, Depends(db_conn)],
     current_user: Annotated[AuthUser, Depends(login_required)],
 ) -> ProjectUserDict:
     """A project manager for a specific project."""
+    # NOTE here we get the project manually to avoid warnings before the project
+    # if fully created yet (about odk_token not existing)
+    project = await DbProject.one(db, project_id, warn_on_missing_token=False)
     return await wrap_check_access(
         project,
         db,

src/backend/app/central/central_crud.py

@@ -274,16 +274,16 @@ def get_project_form_xml(
 
 async def append_fields_to_user_xlsform(
     xlsform: BytesIO,
-    form_category: str = "buildings",
-    additional_entities: list[str] = None,
-    existing_id: str = None,
-    new_geom_type: DbGeomType = DbGeomType.POLYGON,
+    form_name: str = "buildings",
+    additional_entities: Optional[list[str]] = None,
+    existing_id: Optional[str] = None,
+    new_geom_type: Optional[DbGeomType] = DbGeomType.POLYGON,
 ) -> tuple[str, BytesIO]:
     """Helper to return the intermediate XLSForm prior to convert."""
     log.debug("Appending mandatory FMTM fields to XLSForm")
     return await append_mandatory_fields(
         xlsform,
-        form_category=form_category,
+        form_name=form_name,
         additional_entities=additional_entities,
         existing_id=existing_id,
         new_geom_type=new_geom_type,
@@ -292,15 +292,15 @@ async def append_fields_to_user_xlsform(
 
 async def validate_and_update_user_xlsform(
     xlsform: BytesIO,
-    form_category: str = "buildings",
-    additional_entities: list[str] = None,
-    existing_id: str = None,
-    new_geom_type: DbGeomType = DbGeomType.POLYGON,
+    form_name: str = "buildings",
+    additional_entities: Optional[list[str]] = None,
+    existing_id: Optional[str] = None,
+    new_geom_type: Optional[DbGeomType] = DbGeomType.POLYGON,
 ) -> BytesIO:
     """Wrapper to append mandatory fields and validate user uploaded XLSForm."""
     xform_id, updated_file_bytes = await append_fields_to_user_xlsform(
         xlsform,
-        form_category=form_category,
+        form_name=form_name,
         additional_entities=additional_entities,
         existing_id=existing_id,
         new_geom_type=new_geom_type,
@@ -315,7 +315,7 @@ async def update_project_xform(
     xform_id: str,
     odk_id: int,
     xlsform: BytesIO,
-    category: str,
+    # osm_category: str,
     odk_credentials: central_schemas.ODKCentralDecrypted,
 ) -> None:
     """Update and publish the XForm for a project.
@@ -324,7 +324,6 @@ async def update_project_xform(
         xform_id (str): The UUID of the existing XForm in ODK Central.
         odk_id (int): ODK Central form ID.
         xlsform (UploadFile): XForm data.
-        category (str): Category of the XForm.
         odk_credentials (central_schemas.ODKCentralDecrypted): ODK Central creds.
 
     Returns: None
@@ -337,6 +336,7 @@ async def update_project_xform(
     xform_obj.createForm(
         odk_id,
         xform_bytesio,
+        # NOTE this variable is incorrectly named and should be form_id
         form_name=xform_id,
     )
     # The draft form must be published after upload

src/backend/app/central/central_deps.py

@@ -21,9 +21,8 @@
 from contextlib import asynccontextmanager
 from io import BytesIO
 from pathlib import Path
-from typing import Optional
 
-from fastapi import File, UploadFile
+from fastapi import UploadFile
 from fastapi.exceptions import HTTPException
 from osm_fieldwork.OdkCentralAsync import OdkDataset, OdkForm
 
@@ -65,8 +64,8 @@ async def get_async_odk_form(odk_creds: ODKCentralDecrypted):
 
 async def validate_xlsform_extension(xlsform: UploadFile):
     """Validate an XLSForm has .xls or .xlsx extension."""
-    file = Path(xlsform.filename)
-    file_ext = file.suffix.lower()
+    filename = Path(xlsform.filename)
+    file_ext = filename.suffix.lower()
 
     allowed_extensions = [".xls", ".xlsx"]
     if file_ext not in allowed_extensions:
@@ -80,11 +79,3 @@ async def validate_xlsform_extension(xlsform: UploadFile):
 async def read_xlsform(xlsform: UploadFile) -> BytesIO:
     """Read an XLSForm, validate extension, return wrapped in BytesIO."""
     return await validate_xlsform_extension(xlsform)
-
-
-async def read_optional_xlsform(
-    xlsform: Optional[UploadFile] = File(None),
-) -> Optional[BytesIO]:
-    """Read an XLSForm, validate extension, return wrapped in BytesIO."""
-    if xlsform:
-        return await validate_xlsform_extension(xlsform)

src/backend/app/db/enums.py

@@ -247,9 +247,9 @@ class XLSFormType(StrEnum, Enum):
     The the value is the user facing form name (e.g. healthcare).
     """
 
-    buildings = "buildings"
+    buildings = "OSM Buildings"
     # highways = "highways"
-    health = "healthcare"
+    health = "OSM Healthcare"
     # toilets = "toilets"
     # religious = "religious"
     # landusage = "landusage"

src/backend/app/db/models.py

@@ -1026,7 +1026,7 @@ class DbProject(BaseModel):
     custom_tms_url: Optional[str] = None
     status: Optional[ProjectStatus] = None
     visibility: Optional[ProjectVisibility] = None
-    xform_category: Optional[str] = None
+    osm_category: Optional[str] = None
     odk_form_id: Optional[str] = None
     xlsform_content: Optional[bytes] = None
     mapper_level: Optional[MappingLevel] = None
@@ -1036,11 +1036,11 @@ class DbProject(BaseModel):
     odk_central_user: Optional[str] = None
     odk_central_password: Optional[str] = None
     odk_token: Optional[str] = None
-    data_extract_type: Optional[str] = None
     data_extract_url: Optional[str] = None
     task_split_dimension: Optional[int] = None
     task_num_buildings: Optional[int] = None
-    new_geom_type: Optional[DbGeomType] = None
+    primary_geom_type: Optional[DbGeomType] = None  # the main geometries surveyed
+    new_geom_type: Optional[DbGeomType] = None  # when new geometries are drawn
     geo_restrict_distance_meters: Optional[PositiveInt] = None
     geo_restrict_force_error: bool = True
     hashtags: Optional[list[str]] = None
@@ -1087,7 +1087,13 @@ def set_odk_credentials_on_project(
         )
 
     @classmethod
-    async def one(cls, db: Connection, project_id: int, minimal: bool = False) -> Self:
+    async def one(
+        cls,
+        db: Connection,
+        project_id: int,
+        minimal: bool = False,
+        warn_on_missing_token: bool = True,
+    ) -> Self:
         """Get project by ID, including all tasks and other details."""
         # Simpler query without additional metadata
         if minimal:
@@ -1231,7 +1237,7 @@ async def one(cls, db: Connection, project_id: int, minimal: bool = False) -> Se
         if db_project is None:
             raise KeyError(f"Project ({project_id}) not found.")
 
-        if db_project.odk_token is None:
+        if warn_on_missing_token and db_project.odk_token is None:
             log.warning(
                 f"Project ({db_project.id}) has no 'odk_token' set. "
                 "The QRCode will not work!"

src/backend/app/db/postgis_utils.py

@@ -774,17 +774,17 @@ def merge_polygons(
         ) from e
 
 
-def get_osm_geometries(form_category, geometry):
+def get_osm_geometries(osm_category, geometry):
     """Request a snapshot based on the provided geometry.
 
     Args:
-        form_category(str): feature category type (eg: buildings).
+        osm_category(str): feature category type (eg: buildings).
         geometry (str): The geometry data in JSON format.
 
     Returns:
         dict: The JSON response containing the snapshot data.
     """
-    config_filename = XLSFormType(form_category).name
+    config_filename = XLSFormType(osm_category).name
     data_model = f"{data_models_path}/{config_filename}.yaml"
 
     with open(data_model, "rb") as data_model_yaml:

src/backend/app/helpers/helper_routes.py

@@ -66,13 +66,13 @@
 
 @router.get("/download-template-xlsform")
 async def download_template(
-    category: XLSFormType,
+    form_type: XLSFormType,
 ):
-    """Download an XLSForm template to fill out."""
-    filename = XLSFormType(category).name
-    xlsform_path = f"{xlsforms_path}/{filename}.xls"
+    """Download example XLSForm from FMTM."""
+    form_filename = XLSFormType(form_type).name
+    xlsform_path = f"{xlsforms_path}/{form_filename}.xls"
     if Path(xlsform_path).exists():
-        return FileResponse(xlsform_path, filename="form.xls")
+        return FileResponse(xlsform_path, filename=f"{form_filename}.xls")
     else:
         raise HTTPException(status_code=HTTPStatus.NOT_FOUND, detail="Form not found")