PromptFuzz is an automated tool that generates high-quality fuzz drivers for libraries via a fuzz loop constructed on mutating LLMs' prompts. The fuzz loop of PromptFuzz aims to guide the mutation of LLMs' prompts to generate programs that cover more reachable code and explore complex API interrelationships, which are effective for fuzzing.
PromptFuzz is currently regarded as the leading approach for generating fuzz drivers both in academia and industry. The fuzz drivers generated by PromptFuzz achieved a branch coverage of 40.12% on the tested libraries, which is 1.61x greater than OSS-Fuzz and 1.67x greater than Hopper. Besides, PromptFuzz detected 33 valid security bugs from 49 unique crashes.
- Multiply LLM support: Supports the general LLMs: Codex, Incoder, ChatGPT, and GPT4 (Currently tested on ChatGPT).
- Context-based Prompt: Construct LLM prompts with the automatically extracted library context.
- Powerful Sanitization: The program's syntax, semantics, behavior, and coverage are thoroughly analyzed to sanitize the problematic programs.
- Prioritized Mutation: Prioritizes mutating the library API combinations within LLM's prompts to explore complex interrelationships, guided by code coverage.
- Fuzz Driver Exploitation: Infers API constraints using statistics and extends fixed API arguments to receive random bytes from fuzzers.
- Fuzz engine integration: Integrates with grey-box fuzz engine: LibFuzzer.
The fuzz drivers generated by PromptFuzz can detect a wide range of bugs, most of which are security bugs. For instances, CVE-2023-6277, CVE-2023-52355 and CVE-2023-52356.
PromptFuzz detects uniquely interesting bugs:
| ID | Library | Buggy Function | Bug Type | Status | Track Link |
|---|---|---|---|---|---|
| 1. | libaom | highbd_8_variance_sse2 | SEGV | Confirmed | 3489 |
| 2. | libaom | av1_rc_update_framerate | Uninitialized Stack | Confirmed | 3509 |
| 3. | libaom | timebase_units_to_ticks | Integer Overflow | Confirmed | 3510 |
| 4. | libaom | encode_without_recode | SEGV | Confirmed | 3534 |
| 5. | libvpx | vp8_peek_si_internal | SEGV | Confirmed | 1817 |
| 6. | libvpx | update_fragments | Buffer Overflow | Confirmed | 1827 |
| 7. | libvpx | vp8e_encode | Integer Overflow | Confirmed | 1828 |
| 8. | libvpx | encode_mb_row | Integer Overflow | Confirmed | 1831 |
| 9. | libvpx | vpx_free_tpl_gop_stats | SEGV | Confirmed | 1837 |
| 10. | libmagic | apprentice_map | Buffer Overflow | Waiting | 481 |
| 11. | libmagic | magic_setparam | Buffer Overflow | Waiting | 482 |
| 12. | libmagic | check_buffer | Buffer Overflow | Confirmed | 483 |
| 13. | libmagic | mget | Integer Overflow | Waiting | 486 |
| 14. | libTIFF | TIFFOpen | OOM | Confirmed | 614 |
| 15. | libTIFF | PixarLogSetupDecode | OOM | Confirmed | 619 |
| 16. | libTIFF | TIFFReadEncodedStrip | OOM | Confirmed | 620 |
| 17. | libTIFF | TIFFReadRGBAImageOriented | OOM | Confirmed | 620 |
| 18. | libTIFF | TIFFRasterScanlineSize64 | OOM | Confirmed | 621 |
| 19. | libTIFF | TIFFReadRGBATileExt | SEGV | Confirmed | 622 |
| 20. | sqlite3 | sqlite3_unlock_notify | Null Pointer crash | Confirmed | e77a5 |
| 21. | sqlite3 | sqlite3_enable_load_extension | Null Pointer crash | Confirmed | 9ce83 |
| 22. | sqlite3 | sqlite3_db_config | Null Pointer crash | Confirmed | 5e3fc |
| 23. | c-ares | config_sortlist | Memory Leak | Confirmed | d62627 |
| 24. | c-ares | config_sortlist | Memory Leak | Confirmed | d62627 |
| 25. | libjpeg-turbo | tj3DecodeYUV8 | Integer Overflow | Confirmed | 78eaf0 |
| 26. | libjpeg-turbo | tj3LoadImage16 | OOM | Confirmed | 735 |
| 27. | libpcap | pcap_create | File Leak | Confirmed | 1233 |
| 28. | libpcap | pcapint_create_interface | Null Pointer crash | Confirmed | 1239 |
| 29. | libpcap | pcapint_fixup_pcap_pkthdr | Misaligned Address | Confirmed | - |
| 30. | cJSON | cJSON_SetNumberHelper | Error Cast | Confirmed | 805 |
| 31. | cJSON | cJSON_CreateNumber | Error Cast | Confirmed | 806 |
| 32. | cJSON | cJSON_DeleteItemFromObjectCaseSensitive | TimeOut | Confirmed | 807 |
| 33. | curl | parseurl | Assertion Failure | Confirmed | 12775 |
See in Usage
- Close-source libraries: Apply PromptFuzz to close-source libraries by fine tuning LLMs on private code corpus.
- Generalization: Generalize PromptFuzz to binary programs.