Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing diverse v2 crashes #179

Merged
merged 46 commits into from
Jan 26, 2025
Merged
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
efdeb05
postprocess: corrected some smells
equals215 Jan 15, 2025
fd6125f
postprocess: renamed some variables and corrected forloop variables
equals215 Jan 15, 2025
d23a2af
postprocess: postprocessItem args
equals215 Jan 15, 2025
0bac91f
postprocess: never set the state of the parent before adding a child,…
equals215 Jan 15, 2025
2c00d3d
item: reinforced CheckConsistency method
equals215 Jan 15, 2025
55be18c
global: enforcing stricter state and consistency check for items thro…
equals215 Jan 15, 2025
4c0b84d
Merge branch 'dev/v2' into fix/diverse-v2-crash
equals215 Jan 15, 2025
2b74fce
item: corrected CheckConsistency() and made more unit tests
equals215 Jan 15, 2025
283e550
item&finisher: make use of CompleteAndCheck() method on an item to pa…
equals215 Jan 15, 2025
2a5485c
item: CompleteAndCheck() overlooked return conditions
equals215 Jan 15, 2025
4cc8b43
pre/postprocess: trying to fix the flow of childs
equals215 Jan 16, 2025
b69245b
dumper: add a Dump() function to properly dump an Item for further de…
equals215 Jan 16, 2025
2f0a38e
preprocessor: correct exclusion logic
equals215 Jan 16, 2025
4f9c8ef
item.Dedupe: corrected an edge case where a completed child has the s…
equals215 Jan 16, 2025
608ac02
postprocess: correct failed outlink extraction behaviour
equals215 Jan 16, 2025
8f25ee3
Add more detailed pyroscope information
NGTmeaty Jan 14, 2025
8ac78c6
postprocess: add more debug logging to troubleshoot an unknown bug
equals215 Jan 16, 2025
2a68916
preprocess: add itemId in panic
equals215 Jan 16, 2025
14e1e25
postprocess: always postprocess an item EVEN IF ASSETS CAPTURE IS DIS…
equals215 Jan 16, 2025
2c03cf2
archiver: close spooledBuffer if error happened during body processing
equals215 Jan 16, 2025
a121e17
postprocess: close all bodies of an item tree before continuing in th…
equals215 Jan 17, 2025
48ffbe5
archiver: try to write bodies only on disk
equals215 Jan 17, 2025
58fa367
add: small memory optimization for URLToString & encodeQuery
CorentinB Jan 17, 2025
eb09e8c
chore: upgrade Go version & dependencies
CorentinB Jan 17, 2025
4de2bf7
chore: bump warc lib to v.0.8.62
CorentinB Jan 17, 2025
a141e2f
fix: usage of spooledtempfile lib
CorentinB Jan 17, 2025
47bda5e
chore: bump warc lib to v.0.8.63
CorentinB Jan 21, 2025
ec694d5
postprocess: defer a closeBodies call on every item that goes through
equals215 Jan 22, 2025
1c39903
log: disable log queue full error message when TUI is used
equals215 Jan 22, 2025
77ca1ce
cmd: add no-stderr-log flag
equals215 Jan 22, 2025
bf8a08b
hq.consumer: replace previousBatch check with a reactor duplicate check
equals215 Jan 23, 2025
ea9409f
pyroscope: bump upload rate from 15s to 5s
equals215 Jan 23, 2025
d4a7a42
fix: add panic for errors in startPipeline, retry indefinitely on HQ …
CorentinB Jan 23, 2025
92c4ef8
fix: not returning when hq.Start fails to init HQ client
CorentinB Jan 23, 2025
8332d63
fix: typo
CorentinB Jan 23, 2025
b6ec93e
fix: HQ Start failure marking init as already done
CorentinB Jan 23, 2025
5f89e7f
fix: panic when HQ init fails
CorentinB Jan 23, 2025
ad5465b
add: truthsocial.com preprocessing & post-processing
CorentinB Jan 24, 2025
ae6965d
chore: bump warc lib to v.0.8.64
CorentinB Jan 24, 2025
4c56034
add: more truthsocial.com special handling
CorentinB Jan 26, 2025
be2b7b9
add: more truthsocial.com special handling
CorentinB Jan 26, 2025
d445f6d
add: more truthsocial.com special handling
CorentinB Jan 26, 2025
ce0d399
fix: variable scope for truthsocial special handling
CorentinB Jan 26, 2025
1e582f3
fix: domains crawl
CorentinB Jan 26, 2025
5a48d6b
fix: set assets hops to their seed hop
CorentinB Jan 26, 2025
cad639f
fix: extraction of outlinks on assets
CorentinB Jan 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: variable scope for truthsocial special handling
CorentinB committed Jan 26, 2025
commit ce0d39935843aa8c060c7f7a6c4c8210eb64ca24
2 changes: 1 addition & 1 deletion internal/pkg/postprocessor/outlinks.go
Original file line number Diff line number Diff line change
@@ -33,7 +33,7 @@ func extractOutlinks(item *models.Item) (outlinks []*models.URL, err error) {
return outlinks, err
}
case truthsocial.IsAccountLookupURL(item.GetURL()):
outlinks, err := truthsocial.GenerateOutlinksURLsFromLookup(item.GetURL())
outlinks, err = truthsocial.GenerateOutlinksURLsFromLookup(item.GetURL())
if err != nil {
logger.Error("unable to extract outlinks from TruthSocial", "err", err.Error(), "item", item.GetShortID())
return outlinks, err