fix: optimized the directory traversal in check-edit-links script #3590
Conversation
sahitya-chandra
requested review from
derberg,
akshatnema,
magicmatatjahu,
anshgoyalevil,
Mayaleeeee and
asyncapi-bot-eve
as code owners
WalkthroughThe pull request introduces a refactored implementation of directory traversal in the Changes
Assessment against linked issues
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
✨ Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
✅ Deploy Preview for asyncapi-website ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify site configuration. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3590 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 20 20
Lines 732 736 +4
=========================================
+ Hits 732 736 +4 ☔ View full report in Codecov by Sentry. |
|
⚡️ Lighthouse report for the changes in this PR:
Lighthouse ran on https://deploy-preview-3590--asyncapi-website.netlify.app/ |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
scripts/markdown/check-edit-links.js (3)
102-121: Consider performance and compatibility implications of generator usage.While the generator implementation is clean and maintainable, there are some concerns:
- The regenerator-runtime dependency might be too heavyweight
- Sequential
awaitcalls in the loop could impact performance- Memory management for large directories isn't addressed
Consider this alternative implementation using Promise.all for better performance:
-async function* walkDirectory(dir) { +async function walkDirectory(dir) { try { const files = await fs.readdir(dir); - for (const file of files) { - if (file === '_section.md') continue; - - const filePath = path.join(dir, file); - const stats = await fs.stat(filePath); - - if (stats.isDirectory()) { - yield* walkDirectory(filePath); - } else if (stats.isFile() && file.endsWith('.md')) { - yield filePath; - } - } + const entries = await Promise.all( + files + .filter(file => file !== '_section.md') + .map(async file => { + const filePath = path.join(dir, file); + const stats = await fs.stat(filePath); + return { filePath, stats }; + }) + ); + + const results = await Promise.all( + entries.map(async ({ filePath, stats }) => { + if (stats.isDirectory()) { + return walkDirectory(filePath); + } + return stats.isFile() && filePath.endsWith('.md') ? [filePath] : []; + }) + ); + + return results.flat(); } catch (err) { - throw new Error(`Error processing directory ${dir}: ${err.message}`); + throw new Error(`Failed to process directory ${dir}: ${err.code === 'ENOENT' ? 'Directory not found' : err.message}`); } }🧰 Tools
🪛 eslint
[error] 103-104: Delete
⏎(prettier/prettier)
[error] 106-117: iterators/generators require regenerator-runtime, which is too heavyweight for this guide to allow them. Separately, loops should be avoided in favor of array iterations.
(no-restricted-syntax)
[error] 107-107: Unexpected use of continue statement.
(no-continue)
[error] 108-108: Delete
······(prettier/prettier)
[error] 110-110: Unexpected
awaitinside a loop.(no-await-in-loop)
[error] 111-111: Delete
······(prettier/prettier)
123-136: Enhance error handling and type validation.The function is more concise, but could benefit from improved robustness:
- Add type validation for editOptions
- Enhance error handling with specific error types
Consider these improvements:
async function generatePaths(folderPath, editOptions) { + if (!Array.isArray(editOptions)) { + throw new TypeError('editOptions must be an array'); + } + if (!folderPath || typeof folderPath !== 'string') { + throw new TypeError('folderPath must be a non-empty string'); + } const result = []; try { for await (const filePath of walkDirectory(folderPath)) { const relativePath = path.relative(folderPath, filePath); const urlPath = relativePath.split(path.sep).join('/').replace('.md', ''); result.push({ filePath, urlPath, editLink: determineEditLink(urlPath, filePath, editOptions) }); } return result; } catch (err) { - throw new Error(`Error processing directory ${folderPath}: ${err.message}`); + if (err.code === 'ENOENT') { + throw new Error(`Directory not found: ${folderPath}`); + } + throw new Error(`Failed to generate paths: ${err.message}`); } }🧰 Tools
🪛 eslint
[error] 124-125: Delete
⏎··(prettier/prettier)
[error] 127-136: iterators/generators require regenerator-runtime, which is too heavyweight for this guide to allow them. Separately, loops should be avoided in favor of array iterations.
(no-restricted-syntax)
[error] 130-130: Delete
······(prettier/prettier)
102-136: Consider memory optimization for large directory structures.The current implementation loads all file paths into memory. For large directory structures, this could lead to high memory usage.
Consider these memory optimization strategies:
- Implement batch processing:
async function* batchProcessor(generator, batchSize = 1000) { let batch = []; for await (const item of generator) { batch.push(item); if (batch.length >= batchSize) { yield batch; batch = []; } } if (batch.length > 0) { yield batch; } }
- Process results in chunks:
async function generatePaths(folderPath, editOptions) { const batchSize = 1000; const results = []; for await (const batch of batchProcessor(walkDirectory(folderPath), batchSize)) { const processedBatch = batch.map(filePath => { const relativePath = path.relative(folderPath, filePath); const urlPath = relativePath.split(path.sep).join('/').replace('.md', ''); return { filePath, urlPath, editLink: determineEditLink(urlPath, filePath, editOptions) }; }); results.push(...processedBatch); // Optional: Process each batch independently // await processBatch(processedBatch); } return results; }🧰 Tools
🪛 eslint
[error] 103-104: Delete
⏎(prettier/prettier)
[error] 106-117: iterators/generators require regenerator-runtime, which is too heavyweight for this guide to allow them. Separately, loops should be avoided in favor of array iterations.
(no-restricted-syntax)
[error] 107-107: Unexpected use of continue statement.
(no-continue)
[error] 108-108: Delete
······(prettier/prettier)
[error] 110-110: Unexpected
awaitinside a loop.(no-await-in-loop)
[error] 111-111: Delete
······(prettier/prettier)
[error] 124-125: Delete
⏎··(prettier/prettier)
[error] 127-136: iterators/generators require regenerator-runtime, which is too heavyweight for this guide to allow them. Separately, loops should be avoided in favor of array iterations.
(no-restricted-syntax)
[error] 130-130: Delete
······(prettier/prettier)
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
scripts/markdown/check-edit-links.js(1 hunks)
🧰 Additional context used
🪛 eslint
scripts/markdown/check-edit-links.js
[error] 103-104: Delete ⏎
(prettier/prettier)
[error] 106-117: iterators/generators require regenerator-runtime, which is too heavyweight for this guide to allow them. Separately, loops should be avoided in favor of array iterations.
(no-restricted-syntax)
[error] 107-107: Unexpected use of continue statement.
(no-continue)
[error] 108-108: Delete ······
(prettier/prettier)
[error] 110-110: Unexpected await inside a loop.
(no-await-in-loop)
[error] 111-111: Delete ······
(prettier/prettier)
[error] 124-125: Delete ⏎··
(prettier/prettier)
[error] 127-136: iterators/generators require regenerator-runtime, which is too heavyweight for this guide to allow them. Separately, loops should be avoided in favor of array iterations.
(no-restricted-syntax)
[error] 130-130: Delete ······
(prettier/prettier)
⏰ Context from checks skipped due to timeout of 180000ms (1)
- GitHub Check: Lighthouse CI
|
@sahitya-chandra Can you please showcase the performance enhancement of the script, by running the workflow in your forked repo? |
|
before:
after:
@akshatnema sir first approach is better for small directory because it processes all the files parallely which is less memory efficient but, benefits of new approach:
|
Description
Related issue(s)
fixes #3586
Summary by CodeRabbit