@Outlinevpn Process Daily usage statistics

[koechkevin]

Description

Currently, Outline vpn manager statistics are generated monthly by running this script.

dist/cfa_vpn.py/outline_manager.pex -e

We are currently proposing a solution that allows a user get these stats easily without always having the intervention of tech team devs.

Here is the proposed implementation plan

This PR introduces 2 API endpoints.

1. POST /api/userStatistics: To query outline vpn API for the statistics, calculate daily usage then store it in a local database for analysis. This endpoint will be executed via a cron.
2. GET /api/userStatistics: Gets the stored statistics in the database as per the queries sent.

Process Daily User Statistics.

This action is triggered by POST /api/userStatistics. Since The Outline VPN API returns only userId and total data transferred by the user over time, we need to find a way to calculate daily usage by finding the difference from last fetch.

1. Get data transferred from Outline VPN API.
2. Get users from Outline API to get user data from user IDs.
3. Calculate the data transferred by each user. We achieve this by querying the previously fetched and stored cumulative data transfer and subtracting it from the current cumulative obtained from 1 above. This cron will be executed at least once a day at around 11 PM so every calculation will be computed to the day it was run.
4. Store the computed data in the database. If a computation for the current date for the user ID already exists, we simply update it else create a new entry.
5. Return the processed records.

Get statistics.

GET /api/userStatistics
This API allows users to fetch stored statistics from the database based on various filters. The records can be filtered by date, email, date range, user ID, and record ID. Results can also be grouped and ordered based on specific fields.

Parameter	Type	Description	Example
date	string	Filter results by a specific date in YYYY-MM-DD format.	?date=2024-09-03
email	string	Filter results by the user’s email address.	[email protected]
dateBetween.start	string	Start of the date range for filtering. Requires dateBetween.end.	?dateBetween.start=2024-09-03
dateBetween.end	string	End of the date range for filtering. Requires dateBetween.start.	?dateBetween.end=2024-09-04
ID	number	Filter results by a specific record ID.	?ID=45
userId	number	Filter results by a specific user ID.	?userId=107
orderBy	string	Order results by a specific field in either ascending (ASC) or descending (DESC) order.	?orderBy=usage DESC
groupBy *	string	Group results by a specific field. Common options are email, date.	?groupBy=email

The response will return records in the format

[{
        "ID": 103,
        "userId": "107",
        "usage": 9899671,
        "date": "2024-9-3",
        "cumulativeData": 5958400566,
        "email": "[email protected]",
        "createdAt": "2024-09-03T13:07:43.464Z"
    }]

Grouped by Email

[{
        "email": "[email protected]",
        "userId": "107",
        "totalUsage": 9985821081
    }]

Grouped by Date

[{
        "date": "2024-9-4",
        "totalUsage": 21785853001
    }]

Fixes #875

Type of change

• New feature (non-breaking change which adds functionality)

• This change requires a documentation update

Screenshots

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation

[koechkevin]

@m453h refer to this when reviewing implementation plan.

[kilemensi]

👍🏽

--

Think about naming and structuring things e.g. in lib, may be there should be an outline.js file that handles pulling data from the VPN backend. API end points, may be they can be a bit more RESTful i.e. /api/statics with PUT or POST updating the stats and GET returning the stats, etc.

Update 1
On APIs, I think the processGSheet is the biggest offender ...

[m453h]

🚀

Thanks @koechkevin , I've gained further context after carefully reading through the implementation plan and it looks solid. I could be missing something but looking at the Table, I was thinking that UPSERT could simplify the whole operation of calculating and inserting the user stats if we keep userId and date as unique constraints.

[kilemensi]

The key (accessUrl) you posted on above was real and working @koechkevin . I had to delete it for obvious reasons ... Hopeful you'll be able to create a new one.

I think lets remove accessUrl completely from all stats functionality. The only time we should access it is when we need to email it to the user AND it must never be stored or logged anywhere outside the VPN itself.

[github-actions]

Latest updated Preview URL

Name	Review
codeforafrica-ui-pr-870	Visit

[koechkevin]

The data usage API was hardcoded only to obtain data transferred in the last (30 * 24) hours creating a sliding window problem, which makes the daily computations we have proposed in this PR inaccurate.

We may have to open a PR for the above fix or fork the repository to deploy.

[kilemensi]

The data usage API was hardcoded only to obtain data transferred in the last (30 * 24) hours creating a sliding window problem, which makes the daily computations we have proposed in this PR inaccurate.

We may have to open a PR for the above fix or fork the repository to deploy.

Not sure I fully follow @koechkevin (or have full context) but:

1. Haven't we known this all along i.e. isn't this how the current script works @thepsalmist ?
2. Why would we need to fork the repo or another PR?

[koechkevin]

@m453h / @kelvinkipruto

[m453h]

LGTM 🚀

Just an additional thought, the Filters interface could have stricter types for orderBy since we know the fields we would expect