Feedly/fetch reports as incidents

[Mathieu4141]

Contributing to Cortex XSOAR Content

Make sure to register your contribution by filling the contribution registration form

The Pull Request will be reviewed only after the contribution registration form is filled.

Status

• In Progress
• Ready
• In Hold - (Reason for hold)

Description

• Add a new integration to ingest articles as incidents. This was requested by one of our customer to fit their workflow, and discussed with our partner at XSOAR.
• Added a playbook that search and associates some indicators (malware, intrusion sets and TTPs) to the incident.
• Making it optional to ingest articles as indicators in the old feed, to avoid duplicating information in different places if the incident integration is enabled
• Fixed an issue with the continuation that caused some articles to not be ingested if the integration velocity was too high

[content-bot]

Thank you for your contribution. Your generosity and caring are unrivaled! Make sure to register your contribution by filling the Contribution Registration form, so our content wizard @YairGlik will know the proposed changes are ready to be reviewed.
For your convenience, here is a link to the contributions SLAs document.

[content-bot]

Hi @Mathieu4141, thanks for contributing to the XSOAR marketplace. To receive credit for your generous contribution please follow this link.

[YairGlick]

Hi @Mathieu4141,
Thank you for your contribution.
I have not received a response from you in the last two weeks, so I will close your PR now.
Please feel free to re-open it when you are available to continue.
Thanks again.

[Mathieu4141]

Sorry for the late reply @YairGlik
I separated the packs & incident fields as requested

[YairGlick]

Packs/FeedlyArticles/IncidentFields/incident_feedlycrawleddate.json: [BA102] - The given file is not supported in the validate command, see the error above.
The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.
Packs/FeedlyArticles/IncidentTypes/customIncidentTypes.json: [BA102] - The given file is not supported in the validate command, see the error above.
The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.
Packs/FeedFeedly/Integrations/FeedFeedly/FeedFeedly.yml: [BA102] - The given file is not supported in the validate command, see the error above.
The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.
Packs/FeedlyArticles/Classifiers/classifier-Feedly_-_Report_Mapper.json: [BA102] - The given file is not supported in the validate command, see the error above.
The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.
Packs/FeedlyArticles/IncidentFields/incident_feedlythreatactornames.json: [BA102] - The given file is not supported in the validate command, see the error above.
The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.
Packs/FeedlyArticles/IncidentFields/incident_feedlymalwarenames.json: [BA102] - The given file is not supported in the validate command, see the error above.
The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.
Packs/FeedlyArticles/IncidentFields/incident_feedlyurl.json: [BA102] - The given file is not supported in the validate command, see the error above.
The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.
Packs/FeedlyArticles/Integrations/IncidentsFeedly/IncidentsFeedly.yml: [BA102] - The given file is not supported in the validate command, see the error above.
The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.
Packs/FeedlyArticles/Playbooks/Feedly_threats.yml: [BA102] - The given file is not supported in the validate command, see the error above.
The validate command supports: Integrations, Scripts, Playbooks, Incident fields, Incident types, Indicator fields, Indicator types, Objects fields, Object types, Object modules, Images, Release notes, Layouts, Jobs, Wizards, Descriptions And Modeling Rules.

It seems there are issues with several files. Please run demisto-sdk format to fix them.

[Mathieu4141]

It seems there are issues with several files. Please run demisto-sdk format to fix them.

How does it work? I get Did not find any files to format

[YairGlick]

It seems there are issues with several files. Please run demisto-sdk format to fix them.

How does it work? I get Did not find any files to format

for example:
demisto-sdk format -i 'Packs/FeedlyArticles/IncidentTypes/customIncidentTypes.json'

[YairGlick]

please add file Packs/FeedlyArticles/.pack-ignore

[idovandijk]

@Mathieu4141 for me to review I would need all files to be formatted (playbooks included). Feel free to ping me once you're able to format them and I'll take a look!

[Mathieu4141]

@Mathieu4141 for me to review I would need all files to be formatted (playbooks included). Feel free to ping me once you're able to format them and I'll take a look!

Is it good now @idovandijk ?

Also why are the exports invalid in the first place? I downloaded from the XSOAR instance, I'd expect them to be formatted correctly?

[idovandijk]

@Mathieu4141 for me to review I would need all files to be formatted (playbooks included). Feel free to ping me once you're able to format them and I'll take a look!

Is it good now @idovandijk ?

Also why are the exports invalid in the first place? I downloaded from the XSOAR instance, I'd expect them to be formatted correctly?

@Mathieu4141 I can see why this is confusing. Exported files are valid, but they're not release-ready. Using the SDK to format the files ensures they meet the Cortex standard required to work optimally and correctly in the Marketplace.
Formatting the files also allows us to perform some parts of the review that without it we cannot do (we would need to change your files which we would like to avoid). I hope this answers the question. If you're unsure about certain parts of the Format command you can visit https://github.com/demisto/demisto-sdk/blob/master/demisto_sdk/commands/format/README.md

I'll be looking into your updates and let you know if anything else is needed, thanks again

[idovandijk]

Great job, happy to see these new additions for Feedly!

My comments are numbered below for easy reference

Playbook - Feedly threats

1. Please add a data validation task - verify that the MITRE IDs exist in ${incident} before searching it. Same for the rest of the indicators searched. Imagine if one of them ends up empty and we end up querying all indicators and associating all of them to the incident ;)
2. Please add a task that verifies that the search yielded any results, before associating to the incident.

Incident Fields

3. Please set the incident fields to be associated with the Feedly Report incident type instead of being Associated To All.
4. Do you need the fields to be searchable in XSOAR/XSIAM? For example is someone going to search incidents by the Feedly URL field? If not, please set unsearchable to true.

Mapper

5. Once your fields are associated to the Feedly Report incident type, I suggest modifying the mapper to have mapping for the Feedly Report type (not under Common Mapping)

This will keep things more tidy as we currently only intend to map fields related to Feedly Reports.

As always, if there's any question or issue, don't hesitate to reach out to us.
When you're done making changes, let me know and I'll gladly approve from my side.

[Mathieu4141]

Thanks @idovandijk for the explanation and the detailed review!

1 & 2: I'm not sure how to do any of the 2 modifications in the playbook. Which tasks / changes do you suggest to do that?

3, 4, & 5: done :)

[idovandijk]

Thanks @idovandijk for the explanation and the detailed review!

1 & 2: I'm not sure how to do any of the 2 modifications in the playbook. Which tasks / changes do you suggest to do that?

3, 4, & 5: done :)

What I'm suggesting is adding conditional tasks that check that the context you use in your tasks exists prior to executing it.
You can take a look for example at the Extract Indicators From File - Generic v2 playbook, which has a task that checks if a certain file format exists in the context before extracting indicators from the file.

[Mathieu4141]

@idovandijk should be good :)

[idovandijk]

@idovandijk should be good :)

Perfect, that's exactly what I meant. Don't forget to update the playbook image ;)
Adding security approval

[Mathieu4141]

Perfect, that's exactly what I meant. Don't forget to update the playbook image ;)

Great! :D

What do you mean by the playbook image?

[idovandijk]

Perfect, that's exactly what I meant. Don't forget to update the playbook image ;)

Great! :D

What do you mean by the playbook image?

@Mathieu4141 under Packs/your_pack/doc_files
should be a picture of the playbook, which is then used in the Playbook README.
Since you made some changes in the playbook it's a good idea to keep the image up to date too. You can export it from the Playbook Editor using the button on the right side

[Mathieu4141]

I added the image in the folder :)