integrate analyze and fit-score into frontend

* create sign up card * add api/sign_up to postRequests * add api/sign_in to postRequests * change sign in username to email * reformat * remove sign up * create page for task_6 * add sign in * rename sign_up.test.tsx to task_6.test.tsx * fix unclosed brace * add register and login to fetching (#11) * create sign up card * add api/sign_up to postRequests * add api/sign_in to postRequests * change sign in username to email * reformat * remove sign up * create page for task_6 * add sign in * rename sign_up.test.tsx to task_6.test.tsx * add register and login to fetching * fix mismatched braces in fetching.ts * add token to headers in fetching.ts * save token to local storage * fix unable to find element * delete commented code * rename sign up and sign in to register and login respectively * split login and register into separate pages, add input validation, and take field data out of url * add user event library * add testing for login * add testing for register * format * fix eslint errors * fix eslint error * fix localstorage not being defined * fix localstorage not being defined --------- Co-authored-by: julia <[email protected]> * Completed Tasks 4 & 5 with unit tests. (#8) * Completed Tasks 4 & 5 with unit tests. Utilized the utils.test.ts to ensure tasks 4 & 5 unit tests ran correctly. Created a seperate utility file for creating JWT tokens * Changed filenames to snake_case. Deleted deps.ts file. Created function for hashing. Removed the try/catch with the error status 500. * Fixed issue with not being able to read .env files with secret JWT key. Need to use �[0m�[38;5;245mrunning 3 tests from ./api/register_component/user_registration.test.ts�[0m POST /api/register - Successful Registration ... �[0m�[32mok�[0m �[0m�[38;5;245m(6ms)�[0m POST /api/register - Missing Fields ... �[0m�[32mok�[0m �[0m�[38;5;245m(0ms)�[0m POST /api/register - Duplicate Email ... �[0m�[32mok�[0m �[0m�[38;5;245m(3ms)�[0m �[0m�[38;5;245mrunning 1 test from ./api/user_login_component/user_login.test.ts�[0m POST /api/userLogin - Successful Login ... �[0m�[32mok�[0m �[0m�[38;5;245m(7ms)�[0m �[0m�[38;5;245mrunning 1 test from ./hello.test.ts�[0m hello test ... �[0m�[32mok�[0m �[0m�[38;5;245m(1ms)�[0m �[0m�[38;5;245mrunning 5 tests from ./in_memory/in_memory.test.ts�[0m Generate session ID ... �[0m�[32mok�[0m �[0m�[38;5;245m(1ms)�[0m storeData and retrieveData are successful ... �[0m�[32mok�[0m �[0m�[38;5;245m(0ms)�[0m storeData retrieves null for non-existent sessionId ... �[0m�[32mok�[0m �[0m�[38;5;245m(0ms)�[0m deleteData removes data correctly ... �[0m�[32mok�[0m �[0m�[38;5;245m(0ms)�[0m clearAllData removes all session data ... �[0m�[32mok�[0m �[0m�[38;5;245m(0ms)�[0m �[0m�[38;5;245mrunning 2 tests from ./upload/job_description_upload.test.ts�[0m Job Description - Valid Input ... �[0m�[32mok�[0m �[0m�[38;5;245m(9ms)�[0m Job Description - Exceeds Character Limit ... �[0m�[32mok�[0m �[0m�[38;5;245m(0ms)�[0m �[0m�[38;5;245mrunning 3 tests from ./upload/resume_upload.test.ts�[0m Valid PDF ... �[0m�[32mok�[0m �[0m�[38;5;245m(8ms)�[0m Resume Upload - Invalid File Type ... �[0m�[32mok�[0m �[0m�[38;5;245m(1ms)�[0m Oversized File ... �[0m�[32mok�[0m �[0m�[38;5;245m(31ms)�[0m �[0m�[38;5;245mrunning 0 tests from ./util/util.test.ts�[0m �[0m�[32mok�[0m | 15 passed | 0 failed �[0m�[38;5;245m(448ms)�[0m to allow for reading .env files * Finished task 11 along with tests. Created a folder to hold one of the test PDF's and extracted text for testing. Needed to remove the .env from the .gitignore as the name change to .env.local seemed to cause the file to not be found. Updated the command to run tests. * Ran the deno fmt command. Updated the deno.yml file with the correct testing command. * parse text from File object instead of file path * Updated pre-commit file to match deno test in deno.yml * improve jwt helpers * use json for login and registration endpoints * Trying to modify the routes.ts to use middlware to minimize changes to current working file upload api's. * use token for session data * add test for jwt * salt hashes * Created test file for session_middleware.ts. * test session middleware --------- Co-authored-by: julia <[email protected]> * routing and login/registration integration * add loading spinner & protect routes if not logged in * add more unit tests * cleanup and refactor (#16) * refactor frontend * refactor backend * format * add loading spinner & protect routes if not logged in * add more unit tests * format * add api/fit-score to getRequests * add backend get for fit score to dashboard frontend * adapt dashboard tests to using useB ackendGet * remove redundant sign up and sign in * add isError and message to git-score get request type * add isError and message to page * fix some issues from merging * delete task_6 folder * add testing for stars * add test for empty feedback * add message on screen if no suggestions/feedback * add test for null fit score * add message on screen if fit score is null * remove comment * add temporary mock data for testing page * add canvas and update next.js * resolve issues from merging with pdf generation code * resolve linting problems * fix dep issue * add jest canvas mock * loading state * format * add fit-score and analyze to fetching * change feedback.text to feedback.feedback * adapt dashboard to fit-score and analyze post requests * concatenate fit and analyze feedback * add category to feedback * take analyze out of null check * act error * wrap tests in act * backendPostMock before initialization * fix test * cleanup * error test doesnt pass * fix tests and add null analyze test * dont read analyzeResponse.data if analyzeResponse is null * make token handling more fault tolerant * lint * lint and setLoading() in then() * getting closer * Revert "remove unused nlp stuff" This reverts commit 1167343. * use standard nlp stuff to improve keyword extraction in fit-score * better handling for auth * remove console.log * update backend tests * fix mock data repitition and comment empty list test * add isEmpty * format * fix localstorage error --------- Co-authored-by: julia <[email protected]> Co-authored-by: Stephen Ordway <[email protected]>
AlaraBread · Dec 16, 2024 · 382e200 · 382e200
1 parent 8775895
commit 382e200
Show file tree

Hide file tree

Showing 23 changed files with 3,745 additions and 2,029 deletions.
diff --git a/backend/api/analyze/analyze.ts b/backend/api/analyze/analyze.ts
@@ -10,7 +10,7 @@ export default function (router: Router, sessionMiddleware: Middleware) {
 	);
 }
 
-const MAX_TEXT_LENGTH = 10000;
+export const MAX_TEXT_LENGTH = 10000;
 
 export async function analyzeHandler(
 	analyze: typeof analyzeText,
@@ -21,7 +21,12 @@ export async function analyzeHandler(
 	const sessionData = ctx.state.sessionData as SessionData | null;
 
 	// Validate session data
-	if (!sessionData || !sessionData.resumeText || !sessionData.jobDescription) {
+	if (
+		!sessionData ||
+		sessionData.resumeText == undefined ||
+		sessionData.jobDescription == undefined
+	) {
+		console.log(sessionData);
 		ctx.response.status = 400;
 		ctx.response.body = {
 			isError: true,

diff --git a/backend/api/fit_score/fit_score.test.ts b/backend/api/fit_score/fit_score.test.ts
@@ -12,6 +12,13 @@ Deno.test("/api/fit-score - with keywords", async () => {
 				mustHave: ["missing3", "missing4", "notmissing2"],
 			},
 		})),
+		state: {
+			sessionData: {
+				resumeText: "i have experience with java, sql, and git",
+				jobDescription:
+					"we need someone with java, sql, git, and python experience.",
+			},
+		},
 	});
 	await fitScore(ctx);
 	assertEquals(ctx.response.status, 200);
@@ -73,6 +80,12 @@ Deno.test("/api/fit-score - with empty arrays", async () => {
 				mustHave: [],
 			},
 		})),
+		state: {
+			sessionData: {
+				resumeText: "",
+				jobDescription: "",
+			},
+		},
 	});
 	await fitScore(ctx);
 	assertEquals(ctx.response.status, 200);
@@ -90,6 +103,12 @@ Deno.test("/api/fit-score - malformed request", async () => {
 				mustHave: [],
 			},
 		})),
+		state: {
+			sessionData: {
+				resumeText: "",
+				jobDescription: "",
+			},
+		},
 	});
 	await fitScore(ctx);
 	assertEquals(ctx.response.status, 400);

diff --git a/backend/api/fit_score/fit_score.ts b/backend/api/fit_score/fit_score.ts
@@ -1,4 +1,7 @@
 import { Context, Middleware, Router } from "@oak/oak";
+import { keywords } from "./keyword_extraction/keyword_extraction.ts";
+import { SessionData } from "../../in_memory/in_memory.ts";
+import { MAX_TEXT_LENGTH } from "../analyze/analyze.ts";
 
 export default function (router: Router, sessionMiddleware: Middleware) {
 	router.post("/api/fit-score", sessionMiddleware, fitScore);
@@ -23,6 +26,38 @@ export async function fitScore(ctx: Context) {
 		});
 		return;
 	}
+
+	const sessionData = ctx.state.sessionData as SessionData;
+	// Validate session data
+	if (
+		!sessionData ||
+		sessionData.resumeText == undefined ||
+		sessionData.jobDescription == undefined
+	) {
+		console.log(sessionData);
+		ctx.response.status = 400;
+		ctx.response.body = {
+			isError: true,
+			message: "You must upload a resume and job description.",
+		};
+		return;
+	}
+
+	if (
+		sessionData.resumeText.length > MAX_TEXT_LENGTH ||
+		sessionData.jobDescription.length > MAX_TEXT_LENGTH
+	) {
+		ctx.response.status = 400;
+		ctx.response.body = {
+			isError: true,
+			message: "Resume or Job description too long.",
+		};
+		return;
+	}
+
+	req.resumeKeywords = req.resumeKeywords.concat(
+		await keywords(sessionData.resumeText),
+	);
 	ctx.response.status = 200;
 	ctx.response.body = JSON.stringify({
 		isError: false,

diff --git a/backend/api/fit_score/keyword_extraction/keyword_extraction.ts b/backend/api/fit_score/keyword_extraction/keyword_extraction.ts
@@ -0,0 +1,80 @@
+import natural, { DataRecord } from "@natural";
+import { stopwords } from "./stopwords.ts";
+import { keepWords, specialPatterns } from "./specialwords.ts";
+import { dedup } from "../../../util/util.ts";
+
+const wordnet = new natural.WordNet();
+const tokenizer = new natural.WordTokenizer();
+
+const wnCache: Record<string, DataRecord[]> = {};
+// lookup a word in the wordnet database
+// https://wordnet.princeton.edu/
+async function wnLookup(word: string): Promise<DataRecord[]> {
+	if (Object.hasOwn(wnCache, word)) {
+		return Promise.resolve(wnCache[word]);
+	}
+	// wordnet doesnt use promises :(
+	return await new Promise((resolve, _reject) => {
+		wordnet.lookup(word, (record) => {
+			wnCache[word] = record;
+			resolve(record);
+		});
+	});
+}
+
+export async function keywords(text: string): Promise<string[]> {
+	// add seperators for the special patterns
+	text = " " + text.toLowerCase() + " ";
+	const special: string[] = [];
+	specialPatterns.forEach((pattern) => {
+		pattern.lastIndex = 0;
+		let match = pattern.exec(text);
+		while (match && match.groups) {
+			special.push(match.groups.match);
+			text = text.substring(0, match.index) + " " +
+				text.substring(pattern.lastIndex, text.length);
+			pattern.lastIndex -= match.length;
+			match = pattern.exec(text);
+		}
+	});
+	const tokens = tokenizer.tokenize(text).filter((word) =>
+		!stopwords.has(word)
+	);
+	return dedup(
+		(await Promise.all(
+			tokens.map(async (
+				word,
+			) => ({
+				word,
+				lookup: await wnLookup(word),
+			})),
+		)).map(({ word, lookup }) =>
+			lookup.length == 0
+				? { word, lookup: [{ lemma: word, synsetOffset: 0 }] }
+				: {
+					word,
+					lookup,
+				}
+		).map(({ word, lookup }) => ({
+			word,
+			lookup: lookup.sort((a, b) => a.synsetOffset - b.synsetOffset), // sort meanings by most frequently used
+		})).map(({ word, lookup }) =>
+			keepWords.has(word)
+				? word
+				: /^(?<lemma>.*?)(:?\(.+\))?$/.exec(lookup[0].lemma.toLowerCase())
+					?.groups
+					?.lemma as string
+		)
+			.concat(special),
+	);
+}
+
+export async function match(resumeText: string, jobDescriptionText: string) {
+	const resume = await keywords(resumeText);
+	const jobDescription = await keywords(jobDescriptionText);
+	// number of words in both resume and jobDescription
+	const matched = resume.filter((word) => jobDescription.includes(word)).length;
+	return jobDescription.length > 0
+		? 100 * (matched / jobDescription.length)
+		: 0;
+}
diff --git a/backend/api/fit_score/keyword_extraction/specialwords.ts b/backend/api/fit_score/keyword_extraction/specialwords.ts
@@ -0,0 +1,79 @@
+const seperator = /[ \n\t.;,\(\)\[\]]/;
+// these are special patterns that would otherwise be garbled by the tokenizer and/or stopword list
+export const specialPatterns = [
+	// short programming language names that are in the stopword list
+	/c/,
+	/d/,
+	/c\+\+/,
+	/c\/c\+\+/,
+	/\.net\/c#/,
+	/c#/,
+	/r/,
+	// periods would be split by tokenizer
+	/.*?\.js/,
+	/.*?\.py/,
+	/.*?\.net/,
+	/b\.s\./,
+	/b\.a\./,
+	// multi word phrases would get split up by tokenizer
+	/single page app/,
+	/spring boot/,
+	/google spanner/,
+	/github actions/,
+	/google suite/,
+	/microsoft office/,
+	/database administration/,
+	/systems? administration/,
+	/content management/,
+	/user experience/,
+	/version control/,
+	/back[- ]?end/,
+	/front[- ]?end/,
+	/full[- ]?stack/,
+	/cross[- ]browser[- ]compatibility/,
+	/react testing library/,
+	/millions of requests/,
+	/prompt engineering/,
+	/credit cards?/,
+	/point of sale/,
+	/hardware engineering/,
+	/software configuration management/,
+	/high speed networking/,
+	/tcp\/ip/,
+	/pci[- ]express/,
+	/communication protocols?/,
+	/real[- ]time/,
+	/operating systems?/,
+	/computer science/,
+	/artificial intelligence/,
+].map((pattern) => {
+	return new RegExp(
+		seperator.source + "(?<match>" + pattern.source + ")" + seperator.source,
+		"g",
+	);
+});
+
+// these words should be kept rather than being lemmatized
+// any jargon terms that have more common, but irrelevant meanings should go in here
+export const keepWords = new Set([
+	"git",
+	"spanner",
+	"postman",
+	"vim",
+	"bash",
+	"rust",
+	"python",
+	"java",
+	"transition",
+	"migrating",
+	"spa",
+	"restful",
+	"rest",
+	"api",
+	"android",
+	"jest",
+	"usb",
+	"thunderbolt",
+	"ai",
+	"content",
+]);