[Tech-Debt]: Using a streaming implementation when searching in a file for a regex

[shawn-hurley]

We are worried about memory usage of the binary running an unbounded number of threads reading files implemented here

We also do the same thing in the generic provider, and we should consider moving both to streaming the files.

[Parthiba-Hazra]

@shawn-hurley can I work on it?

[shawn-hurley]

@Parthiba-Hazra

I would not be opposed to trying and getting this working again. To note: we had to revert the change in #430 because of negative look-ahead regex searches and we would need to solve that problem simultaneously.

I do not currently have a good way of doing that, so I worry about this.

It sounds like you are eager to take something and that is genuinely appreciated. Can I work with you on Slack because it is apparent that after the break and the drive for the release we may need to do some issue cleanup.

[Parthiba-Hazra]

@shawn-hurley I see, I will reach out to you on slack, thanks for the clarification.

[shawn-hurley]

Sorry again, I wish that I had something off the top of my head to give you, but I am just getting back into the swing of things after a break so I also need to look through things

[Parthiba-Hazra]

@shawn-hurley no problem I can understand.

[shawn-hurley]

I am considering closing this issue. We attempted this, but it is a problem because golang regex doesn't support negative look-ahead (IIRC). @pranavgaikwad @fabianvf @jmle can you advise?

[pranavgaikwad]

@shawn-hurley Yes, this should be closed. not using go regex anymore