Crawler Overhaul #114
Crawler Overhaul #114
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
frontend: do not show delete button for library papers
…efactor crawler (go)
Crawler -
|
backend/src/bin/library-papers.rs
Outdated
| .await | ||
| .expect("Failed to connect to database"); | ||
|
|
||
| let mut errored = false; |
There was a problem hiding this comment.
Can this be done without a mutable variable?
There was a problem hiding this comment.
not sure (easily), maybe can print success msg for each paper instead of globally?
|
|
||
| if qp.approve_status { | ||
| if let Some(similar) = similar_papers.iter().next() { | ||
| // todo: what if there are multiple similar papers? |
There was a problem hiding this comment.
Has this been resolved?
There was a problem hiding this comment.
Not sure what to do if there are multiple
| let new_path = env_vars.paths.get_path_from_slug(&file_link_slug); | ||
|
|
||
| if let Err(e) = fs::copy(file_path, new_path) { | ||
| eprintln!("Failed to copy file: {}", e); |
There was a problem hiding this comment.
Can this be logged to a logfile instead? Each restore can create a new logfile.
crawler/crawler.go
Outdated
|
|
||
| if len(name_split[0]) == 7 { | ||
| if len(name_split[0]) == 7 && re.MatchString(name_split[0]) { |
There was a problem hiding this comment.
Some old course codes had a length of less than 7 I believe.
There was a problem hiding this comment.
well, the regex checks for 7 char codes. we'll have to change that too
| } | ||
| } | ||
| name = strings.Join(name_split, " ") | ||
| if len(name) < 5 { // assuming course name is at least 5 characters long |
There was a problem hiding this comment.
Is this required if the next check is done anyway?
There was a problem hiding this comment.
unsure what 'next check' refers to. this check (len(name) < 5) is there because- if the file on peqp contains a name which is <5 chars long, very likely it isn't the course name. if the course also doesn't exist on courses.json, we would end up with the same, likely wrong name for the course in the DB. Here, setting it to empty string forces it to be added as unapproved
yeah, its a bit hacky, not sure how to make this clearer
| } | ||
|
|
||
| let (mut tx, id) = database.insert_new_library_qp(&qp).await?; | ||
| let file_name = format!("{}_{}", id, qp.filename); |
There was a problem hiding this comment.
What is qp.filename usually? Maybe it would be better to use the format used for the uploaded papers instead?
There was a problem hiding this comment.
qp.filename would be
2024_End-Spring_[Department]_[CourseCode]_[CourseName]_ES_2024.pdf
basically just the url path on PEQP
…o of routines cli arg
Fixes #91
Fixes #92
Description
courses.json. If not, found, use the rest of the file name as the course name.