Qualcomm AI Engine Direct - Optimize static llama phase 2

[shewu-quic]

• Modify custom annotation for kv in llama
• Remove unneccesary reshape op
• Refactor requantize mechanism
  • Support partial of ops to insert to_copy
  • For the same quant_attr share the same to_copy op
• Remove qdq for I/O
• Refine calibration and runner in prefill mode
• (new) Support graph sharding in hybrid mode

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7466

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 9def8ff with merge base 7a2dc47 ():

NEW FAILURE - The following job has failed:

• Check Labels / Check labels (gh)
  RuntimeError: Error checking labels: PR does not have required labels

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Hey can you help rebase?

[cccclai]

LGTM and thank you for the refactor!

[shewu-quic]

Hey can you help rebase?

I have rebased.

I add graph sharding support in hybrid mode. This has the most significant impact on performance. When HTP execution time exceeds RPC polling time (10 ms in burst mode), CPU frequency may drop to its minimum. Our experiments indicate that setting num_sharding to 4 yields the best results. However, this depends on the device. Aim to keep the execution time for each partition under 10ms ( RPC polling time).

executorch/backends/qualcomm/runtime/backends/htpbackend/HtpDevice.cpp

Line 262 in 6c9b9b6

rpc_polling_time.rpcPollingTimeConfig = kRpcPollingTimeHighPower;

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.