[backend/frontend] add configuration to have better chances to get RSS Feed contents (#8736)

[aHenryJard]

Proposed changes

• Do not set proxyAgent header when there is no proxy configured

Adding several configuration to get better chance to avoid HTTP 403 from public RSS Feed:

• ingestion_manager.rss_feed.min_interval_minutes : wait for at least N minutes before next HTTP call for RSS Feed
• ingestion_manager.rss_feed.user_agent: allow to overide user agent used in header for RSS Feed requests
• ingestion_manager.csv_feed.min_interval_minutes : wait for at least N minutes before next HTTP call for CSV Feed

Adding some info and warn level log:

• Info to know if an HTTP Call has been skipped because of min_interval_minutes or queue states
• warn in case of 403 coming from a cloudflare challenge to know that it's not a credential issue.

Related issues

• closes [RSS Feed] Error 403 on accessible public feeds #8736

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality
• I wrote test cases for the relevant uses case (coverage and e2e)
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality

Further comments

[codecov]

Codecov Report

Attention: Patch coverage is 31.11111% with 31 lines in your changes missing coverage. Please review.

Project coverage is 65.10%. Comparing base (b293e2e) to head (4a7b302).

Files with missing lines	Patch %	Lines
...rm/opencti-graphql/src/manager/ingestionManager.ts	17.14%	29 Missing ⚠️
...-platform/opencti-graphql/src/utils/http-client.ts	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9244      +/-   ##
==========================================
- Coverage   65.12%   65.10%   -0.03%     
==========================================
  Files         630      630              
  Lines       60083    60117      +34     
  Branches     6694     6703       +9     
==========================================
+ Hits        39127    39137      +10     
- Misses      20956    20980      +24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[SouadHadjiat]

why strict > ? so if CSV_FEED_MIN_INTERVAL_MINUTES is 5min, and sinceNowInMinutes(last_execution_date) is 5, it won't execute until the next minute?

[SouadHadjiat]

and since this code is the same for RSS, I suggest to maybe refactor it to use a method like :

const shouldExexcuteIngestion = (ingestion, min_interval_minutes) => {
   const { last_execution_date } = ingestion;
   return !last_execution_date || sinceNowInMinutes(last_execution_date) > min_interval_minutes;
}

so we could call it like this shouldExecuteIngestion(ingestion, CSV_FEED_MIN_INTERVAL_MINUTES) or shouldExecuteIngestion(ingestion, RSS_FEED_MIN_INTERVAL_MINUTES)

[aHenryJard]

Inerval is 30s so it will wait only 30s (well for default values). Anyway I'm adding an <= and ok for shouldExecuteIngestion

[lndrtrbn]

No need for this || '-' check here?

[lndrtrbn]

Not sure to understand the logic behind the changes. Moreover, defaultHttpAgent is always undefined

[aHenryJard]

I'm going to testagain because it's the more risky change, but the issue was that defaultHttpAgent was set to new Agent() instead of undefined. Then is there is some proxy configuration the agent is correctly initialize in

opencti/opencti-platform/opencti-graphql/src/config/conf.js

Lines 396 to 409 in b293e2e

	export const getPlatformHttpProxyAgent = (uri) => {
	const platformProxies = getPlatformHttpProxies();
	const targetUrl = new URL(uri);
	const targetProxy = platformProxies[targetUrl.protocol]; // Select the proxy according to target protocol
	if (targetProxy) {
	// If proxy found, check if hostname is not excluded
	if (targetProxy.isExcluded(targetUrl.hostname)) {
	return undefined;
	}
	// If not generate the agent accordingly
	return targetProxy.build();
	}
	return undefined;
	};

As you see there, if no proxy the return is undefined, which means that in the http call there is no empty httpAgent => from my test having empty httpAgent in request is what is causing being refused by cloudflare.

[aHenryJard]

Maybe I should just remove defaultHttpAgent in http-client