New Feeds API. Closes #97, #98 and #429. #419
Conversation
…apt tests accordingly
…s and remove unused code
|
@mlodic : One thing I forgot: As the new API is only accessible by authenticated users, should we add the option to exclude mass scanners to the old APIs? |
for "mass scanners" you mean legit scanners like Censys, right? if so, yes |
api/serializers.py
Outdated
| @cache | ||
| def ordering_validation(ordering: str) -> str: | ||
| if not ordering: | ||
| raise serializers.ValidationError("Invalid ordering: <empty string>") | ||
| if ordering[0] == "-": | ||
| ordering = ordering[1:] | ||
| try: | ||
| IOC._meta.get_field(ordering) | ||
| except FieldDoesNotExist as exc: | ||
| raise serializers.ValidationError(f"Invalid ordering: {ordering}") from exc | ||
| return ordering |
There was a problem hiding this comment.
I am not completely sure to understand what happens here. You remove the - if exists and then return the field value without it. But isn't it needed to properly order the query afterwards?
There was a problem hiding this comment.
You are right, this is indeed confusing. I will make the function clearer.
| @@ -54,25 +56,54 @@ def feed_type_validation(feed_type: str, valid_feed_types: frozenset) -> str: | |||
| return feed_type | |||
|
|
|||
|
|
|||
| class FeedsSerializer(serializers.Serializer): | |||
| @cache | |||
There was a problem hiding this comment.
is this useful? I think I am missing the point of this function. Could you clarify it?
There was a problem hiding this comment.
The @cache decorator? I assume that the function will very often be called with the same ordering argument, e.g. -last_seen which is the default ordering. If the @cache decorator is used and the function has been called at least once with a given argument, the result of the function will be obtained by a dictionary lookup instead of executing the function. This should be much quicker.
api/views/utils.py
Outdated
| class FeedRequestParams: | ||
| """A class to handle and validate feed request parameters. | ||
| It processes and stores query parameters for feed requests, | ||
| providing default values. | ||
|
|
||
| Attributes: | ||
| feed_type (str): Type of feed to retrieve (default: "all") | ||
| attack_type (str): Type of attack to filter (default: "all") | ||
| max_age (str): Maximum number of days since last occurrence (default: "3") | ||
| min_days_seen (str): Minimum number of days on which an IOC must have been seen (default: "1") | ||
| include_reputation (list): List of reputation values to include (default: []) | ||
| exclude_reputation (list): List of reputation values to exclude (default: []) | ||
| feed_size (int): Number of items to return in feed (default: "5000") | ||
| ordering (str): Field to order results by (default: "-last_seen") | ||
| verbose (str): Whether to include IOC properties that contain a lot of data (default: "false") | ||
| paginate (str): Whether to paginate results (default: "false") | ||
| format (str): Response format type (default: "json") | ||
| """ | ||
|
|
||
| def __init__(self, query_params: dict): | ||
| """Initialize a new FeedRequestParams instance. | ||
|
|
||
| Parameters: | ||
| query_params (dict): Dictionary containing query parameters for feed configuration. | ||
| """ | ||
| self.feed_type = query_params.get("feed_type", "all").lower() | ||
| self.attack_type = query_params.get("attack_type", "all").lower() | ||
| self.max_age = query_params.get("max_age", "3") | ||
| self.min_days_seen = query_params.get("min_days_seen", "1") | ||
| self.include_reputation = query_params["include_reputation"].split(";") if "include_reputation" in query_params else [] | ||
| self.exclude_reputation = query_params["exclude_reputation"].split(";") if "exclude_reputation" in query_params else [] | ||
| self.feed_size = query_params.get("feed_size", "5000") | ||
| self.ordering = query_params.get("ordering", "-last_seen").lower().replace("value", "name") | ||
| self.verbose = query_params.get("verbose", "false").lower() | ||
| self.paginate = query_params.get("paginate", "false").lower() | ||
| self.format = query_params.get("format_", "json").lower() | ||
|
|
||
| def exclude_mass_scanners(self): | ||
| self.exclude_reputation.append("mass scanner") | ||
|
|
||
| def set_legacy_age(self, age: str): | ||
| """Translates legacy age specification into max_age and min_days_seen attributes | ||
| and sets ordering accordingly. | ||
|
|
||
| Parameters: | ||
| age (str): Age of the data to filter (recent or persistent). | ||
| """ | ||
| match age: | ||
| case "recent": | ||
| self.max_age = "3" | ||
| self.min_days_seen = "1" | ||
| if "feed_type" in self.ordering: | ||
| self.ordering = "-last_seen" | ||
| case "persistent": | ||
| self.max_age = "14" | ||
| self.min_days_seen: "10" | ||
| if "feed_type" in self.ordering: | ||
| self.ordering = "-attack_count" |
There was a problem hiding this comment.
looking at this closely, this could be all converted inside the Django Rest Framework but don't worry, I won't ask you to change it 🤣 . The important thing is that you don't find any particular perfomance issues, as I said today. I'll check that in the Honeynet production servers too
There was a problem hiding this comment.
Ok :D If there's the option to do this with a built-in class from Django Rest Framework I would like to use it. Would you show me where to find it?
There was a problem hiding this comment.
But I would like to change that in a separate PR.
…nctions for field-level validation
- remove logging the number of returned IOCs in get_queryset() as this forces early QuerySet evaluation - only pass filter for number_of_days_seen if it is > 1 - build list of general honeypots, that have seen an IOC, directly on the database - rewrite feeds_response() function
regulartim
changed the title
Description
This PR introduces a new API endpoint for feeds and adds some data to the existing ones. Changes in detail:
/api/feeds/v2) that provides highly configurable and more detailed feeds. This endpoint is only accessible for authenticated users.Related issues
Type of change
Checklist
develop.Black,Flake,Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.Important Rules