Setup token analysis for all functions #256

ngundotra · 2025-02-01T22:55:40Z

Add Token Counting Utility

When invoking an LLM with all tools (ie createSolanaTools(agentKit)), all tools are passed every time a function call occurs, consuming a high number of tokens per request. As the number of tools increases, this can lead to breaking context windows, model decoherence, and increased costs.

Here is an example trace from Langsmith showing 40,000 tokens being consumed to answer the question "who owns toly.sol". For context, gpt-4o has a max response size of 16,000 tokens

The goal of this PR is to setup a way to easily profile high token function calls, so developers can effectively pick & choose which tools they want to include in their agents.

Changes Made

This PR adds the following changes:

adds pnpm tool-summary to print out a table of how many tokens each function uses in each respective framework (as a Solana Agent Kit Action & as a Vercel CoreTool)
adds pnpm tool-summary:langchain to print out a table of how many tokens each langchain Tool uses

Example of pnpm tool-summary

$ pnpm tool-summary

> [email protected] tool-summary /home/ubuntu/solana-agent-kit
> tsx src/utils/analyzeTools.ts


Cross-Implementation Tool Token Analysis:
┌─────────┬─────────────────────────────────────────────────────┬───────────────┬──────────────────┐
│ (index) │ Name                                                │ Action Tokens │ Vercel AI Tokens │
├─────────┼─────────────────────────────────────────────────────┼───────────────┼──────────────────┤
│ 0       │ 'APPROVE_MULTISIG_PROPOSAL_ACTION'                  │ 177           │ 205              │
│ 1       │ 'DRIFT_GET_ENTRY_QUOTE_OF_PERP_TRADE_ACTION'        │ 165           │ 201              │
│ 2       │ 'CREATE_MULTISIG_ACTION'                            │ 167           │ 198              │
│ 3       │ 'CREATE_MULTISIG_PROPOSAL_ACTION'                   │ 169           │ 196              │
│ 4       │ 'DEPOSIT_TO_MULTISIG_ACTION'                        │ 148           │ 179              │
│ 5       │ 'REJECT_MULTISIG_PROPOSAL_ACTION'                   │ 150           │ 177              │
│ 6       │ 'EXECUTE_MULTISIG_PROPOSAL_ACTION'                  │ 150           │ 177              │
│ 7       │ 'TRANSFER_FROM_MULTISIG_ACTION'                     │ 144           │ 171              │
│ 8       │ 'LULO_LEND_ACTION'                                  │ 130           │ 178              │
│ 9       │ 'LULO_WITHDRAW_ACTION'                              │ 129           │ 177              │
│ 10      │ 'CREATE_ORCA_CLMM_ACTION'                           │ 141           │ 164              │
│ 11      │ 'DRIFT_SPOT_TOKEN_SWAP_ACTION'                      │ 138           │ 165              │
│ 12      │ 'STAKE_WITH_SOLAYER_ACTION'                         │ 120           │ 170              │
│ 13      │ 'OPEN_ORCA_CENTERED_POSITION_WITH_LIQUIDITY_ACTION' │ 130           │ 157              │
│ 14      │ 'DRIFT_PERP_MARKET_FUNDING_RATE_ACTION'             │ 128           │ 156              │
│ 15      │ 'RAYDIUM_CREATE_AMM_V4_ACTION'                      │ 125           │ 157              │
│ 16      │ 'DRIFT_GET_LEND_AND_BORROW_APY_ACTION'              │ 128           │ 152              │
│ 17      │ 'BALANCE_ACTION'                                    │ 122           │ 149              │
│ 18      │ 'GET_OWNED_DOMAINS_FOR_TLD_ACTION'                  │ 115           │ 146              │
│ 19      │ 'GET_TOKEN_DATA_ACTION'                             │ 116           │ 144              │
│ 20      │ 'CLOSE_ORCA_POSITION_ACTION'                        │ 118           │ 142              │
│ 21      │ 'RAYDIUM_CREATE_CPMM_ACTION'                        │ 111           │ 143              │
│ 22      │ 'GET_PRICE_INFERENCE_ACTION'                        │ 115           │ 139              │
│ 23      │ 'FETCH_ORCA_POSITIONS_ACTION'                       │ 113           │ 136              │
│ 24      │ 'TOKEN_BALANCES_ACTION'                             │ 109           │ 136              │
│ 25      │ 'WITHDRAW_VOLTR_STRATEGY_ACTION'                    │ 107           │ 138              │
│ 26      │ 'CREATE_ORCA_SINGLE_SIDED_WHIRLPOOL_ACTION'         │ 106           │ 138              │
│ 27      │ 'DEPOSIT_VOLTR_STRATEGY_ACTION'                     │ 105           │ 136              │
│ 28      │ 'SWITCHBOARD_FEED_ACTION'                           │ 103           │ 138              │
│ 29      │ 'GET_ALL_DOMAINS_TLDS_ACTION'                       │ 103           │ 135              │
│ 30      │ 'GET_VOLTR_POSITION_VALUES_ACTION'                  │ 102           │ 134              │
│ 31      │ 'WITHDRAW_OR_BORROW_FROM_DRIFT_ACCOUNT_ACTION'      │ 99            │ 135              │
│ 32      │ 'RESOLVE_SOL_DOMAIN_ACTION'                         │ 100           │ 132              │
│ 33      │ 'FLASH_CLOSE_TRADE_ACTION'                          │ 92            │ 136              │
│ 34      │ 'TRADE_DELEGATED_DRIFT_VAULT_ACTION'                │ 94            │ 133              │
│ 35      │ 'LAUNCH_PUMPFUN_TOKEN_ACTION'                       │ 97            │ 129              │
│ 36      │ 'STAKE_WITH_JUP_ACTION'                             │ 96            │ 127              │
│ 37      │ 'PYTH_FETCH_PRICE_ACTION'                           │ 95            │ 127              │
│ 38      │ 'FLASH_OPEN_TRADE_ACTION'                           │ 91            │ 130              │
│ 39      │ 'GET_OWNED_ALL_DOMAINS_ACTION'                      │ 94            │ 126              │
│ 40      │ 'UNSTAKE_FROM_DRIFT_INSURANCE_FUND_ACTION'          │ 100           │ 120              │
│ 41      │ 'GET_PRIMARY_DOMAIN_ACTION'                         │ 93            │ 125              │
│ 42      │ 'GET_ALL_TOPICS_ACTION'                             │ 93            │ 125              │
│ 43      │ 'CREATE_IMAGE_ACTION'                               │ 92            │ 124              │
│ 44      │ 'GET_ALL_REGISTERED_ALL_DOMAINS_ACTION'             │ 92            │ 124              │
│ 45      │ 'GET_ASSETS_BY_AUTHORITY_ACTION'                    │ 96            │ 119              │
│ 46      │ 'GET_ASSETS_BY_CREATOR_ACTION'                      │ 96            │ 119              │
│ 47      │ 'CREATE_GIBWORK_TASK_ACTION'                        │ 91            │ 123              │
│ 48      │ 'GET_MAIN_ALL_DOMAINS_DOMAIN_ACTION'                │ 91            │ 123              │
│ 49      │ 'LEND_ASSET_ACTION'                                 │ 91            │ 122              │
│ 50      │ 'SEND_TRANSACTION_WITH_PRIORITY_ACTION'             │ 97            │ 115              │
│ 51      │ 'OPEN_ORCA_SINGLE_SIDED_POSITION_ACTION'            │ 95            │ 115              │
│ 52      │ 'GET_ASSETS_BY_OWNER_ACTION'                        │ 91            │ 119              │
│ 53      │ 'REQUEST_UNSTAKE_FROM_DRIFT_INSURANCE_FUND_ACTION'  │ 97            │ 113              │
│ 54      │ 'RESOLVE_DOMAIN_ACTION'                             │ 88            │ 120              │
│ 55      │ 'GET_TPS_ACTION'                                    │ 90            │ 118              │
│ 56      │ 'FETCH_PRICE_ACTION'                                │ 90            │ 118              │
│ 57      │ 'REGISTER_DOMAIN_ACTION'                            │ 88            │ 120              │
│ 58      │ 'GET_INFO_ACTION'                                   │ 88            │ 119              │
│ 59      │ 'DEPOSIT_TO_DRIFT_USER_ACCOUNT_ACTION'              │ 91            │ 115              │
│ 60      │ 'TRADE_ACTION'                                      │ 89            │ 116              │
│ 61      │ 'REQUEST_FUNDS_ACTION'                              │ 89            │ 116              │
│ 62      │ 'AVAILABLE_DRIFT_MARKETS_ACTION'                    │ 89            │ 116              │
│ 63      │ 'DEPLOY_TOKEN_ACTION'                               │ 87            │ 115              │
│ 64      │ 'DOES_USER_HAVE_DRIFT_ACCOUNT_ACTION'               │ 90            │ 110              │
│ 65      │ 'TRANSFER_ACTION'                                   │ 86            │ 113              │
│ 66      │ 'MINT_NFT_ACTION'                                   │ 86            │ 113              │
│ 67      │ 'SWAP_ACTION'                                       │ 90            │ 109              │
│ 68      │ 'TRADE_DRIFT_PERP_ACCOUNT_ACTION'                   │ 85            │ 113              │
│ 69      │ 'PARSE_TRANSACTION_ACTION'                          │ 87            │ 110              │
│ 70      │ 'DEPLOY_COLLECTION_ACTION'                          │ 84            │ 111              │
│ 71      │ 'GET_INFERENCE_BY_TOPIC_ID_ACTION'                  │ 88            │ 104              │
│ 72      │ 'CREATE_DRIFT_VAULT_ACTION'                         │ 86            │ 105              │
│ 73      │ 'CREATE_WEBHOOK_ACTION'                             │ 85            │ 104              │
│ 74      │ 'GET_ASSET_ACTION'                                  │ 83            │ 106              │
│ 75      │ 'CREATE_DRIFT_USER_ACCOUNT_ACTION'                  │ 82            │ 102              │
│ 76      │ 'GET_WEBHOOK_ACTION'                                │ 82            │ 101              │
│ 77      │ 'UPDATE_DRIFT_VAULT_ACTION'                         │ 81            │ 100              │
│ 78      │ 'WITHDRAW_FROM_DRIFT_VAULT_ACTION'                  │ 83            │ 97               │
│ 79      │ 'DERIVE_DRIFT_VAULT_ADDRESS_ACTION'                 │ 81            │ 97               │
│ 80      │ 'DELETE_WEBHOOK_ACTION'                             │ 78            │ 97               │
│ 81      │ 'DEPOSIT_INTO_DRIFT_VAULT_ACTION'                   │ 78            │ 94               │
│ 82      │ 'REQUEST_WITHDRAWAL_FROM_DRIFT_VAULT_ACTION'        │ 78            │ 94               │
│ 83      │ 'DRIFT_VAULT_INFO_ACTION'                           │ 76            │ 96               │
│ 84      │ 'DRIFT_USER_ACCOUNT_INFO_ACTION'                    │ 77            │ 93               │
│ 85      │ 'UPDATE_DRIFT_VAULT_DELEGATE_ACTION'                │ 77            │ 93               │
│ 86      │ 'WALLET_ADDRESS_ACTION'                             │ 72            │ 92               │
│ 87      │ 'STAKE_TO_DRIFT_INSURANCE_FUND_ACTION'              │ 76            │ 88               │
└─────────┴─────────────────────────────────────────────────────┴───────────────┴──────────────────┘

Category Totals:
┌─────────┬──────────────┬──────────────┬─────────────┐
│ (index) │ Action Total │ Vercel Total │ Grand Total │
├─────────┼──────────────┼──────────────┼─────────────┤
│ 0       │ 9017         │ 11410        │ 20427       │
└─────────┴──────────────┴──────────────┴─────────────┘

Here's example of pnpm tool-summary:langchain:

$ pnpm tool-summary:langchain

> [email protected] tool-summary:langchain /home/ubuntu/solana-agent-kit
> tsx src/utils/analyzeTools.ts --langchain


Cross-Implementation Tool Token Analysis:
┌─────────┬──────────────────────────────────────────────┬──────────────────┐
│ (index) │ Name                                         │ Langchain Tokens │
├─────────┼──────────────────────────────────────────────┼──────────────────┤
│ 0       │ 'solana_batch_order'                         │ 425              │
│ 1       │ 'cross_chain_swap'                           │ 372              │
│ 2       │ 'solana_open_perp_trade'                     │ 371              │
│ 3       │ 'orca_create_single_sided_liquidity_pool'    │ 340              │
│ 4       │ '3land_minting_tool_single'                  │ 340              │
│ 5       │ 'meteora_create_dlmm_pool'                   │ 315              │
│ 6       │ 'meteora_create_dynamic_pool'                │ 301              │
│ 7       │ 'solana_compressed_airdrop'                  │ 291              │
│ 8       │ 'solana_mint_nft'                            │ 284              │
│ 9       │ 'create_helius_webhook'                      │ 277              │
│ 10      │ 'orca_create_clmm'                           │ 271              │
│ 11      │ 'create_drift_vault'                         │ 251              │
│ 12      │ 'create_gibwork_task'                        │ 236              │
│ 13      │ 'solana_limit_order'                         │ 235              │
│ 14      │ 'solana_get_assets_by_creator'               │ 235              │
│ 15      │ 'solana_trade'                               │ 232              │
│ 16      │ 'orca_open_single_sided_position'            │ 226              │
│ 17      │ '3land_minting_tool_collection'              │ 226              │
│ 18      │ 'solana_close_perp_trade'                    │ 226              │
│ 19      │ 'solana_get_assets_by_authority'             │ 225              │
│ 20      │ 'solana_transfer'                            │ 223              │
│ 21      │ 'solana_launch_pumpfun_token'                │ 221              │
│ 22      │ 'solana_balance_other'                       │ 220              │
│ 23      │ 'raydium_create_cpmm'                        │ 218              │
│ 24      │ 'orca_open_centered_position_with_liquidity' │ 216              │
│ 25      │ 'solana_flash_open_trade'                    │ 216              │
│ 26      │ 'update_drift_vault'                         │ 211              │
│ 27      │ 'drift_spot_token_swap'                      │ 198              │
│ 28      │ 'raydium_create_clmm'                        │ 196              │
│ 29      │ 'solana_deploy_token'                        │ 184              │
│ 30      │ 'switchboard_simulate_feed'                  │ 183              │
│ 31      │ 'solana_lulo_withdraw'                       │ 181              │
│ 32      │ 'solana_send_transaction_with_priority_fee'  │ 181              │
│ 33      │ 'trade_delegated_drift_vault'                │ 181              │
│ 34      │ 'solana_balance'                             │ 177              │
│ 35      │ 'solana_lulo_lend'                           │ 176              │
│ 36      │ 'solana_get_all_assets_by_owner'             │ 176              │
│ 37      │ 'trade_drift_perp_account'                   │ 173              │
│ 38      │ 'raydium_create_ammV4'                       │ 172              │
│ 39      │ 'create_2by2_multisig'                       │ 172              │
│ 40      │ 'execute_proposal_2by2_multisig'             │ 172              │
│ 41      │ 'create_proposal_2by2_multisig'              │ 170              │
│ 42      │ 'approve_proposal_2by2_multisig'             │ 170              │
│ 43      │ 'reject_proposal_2by2_multisig'              │ 170              │
│ 44      │ 'solana_resolve_all_domains'                 │ 168              │
│ 45      │ 'solana_cancel_all_orders'                   │ 166              │
│ 46      │ 'solana_withdraw_all'                        │ 166              │
│ 47      │ 'transfer_from_2by2_multisig'                │ 166              │
│ 48      │ 'rock_paper_scissors'                        │ 164              │
│ 49      │ 'solana_flash_close_trade'                   │ 164              │
│ 50      │ 'solana_deploy_collection'                   │ 160              │
│ 51      │ 'orca_close_position'                        │ 157              │
│ 52      │ 'solana_fetch_token_report_summary'          │ 157              │
│ 53      │ 'solana_fetch_token_detailed_report'         │ 157              │
│ 54      │ 'solana_get_asset'                           │ 156              │
│ 55      │ 'solana_fetch_price'                         │ 153              │
│ 56      │ 'solana_get_domain'                          │ 153              │
│ 57      │ 'solana_resolve_domain'                      │ 153              │
│ 58      │ 'withdraw_from_drift_account'                │ 153              │
│ 59      │ 'solana_get_main_domain'                     │ 152              │
│ 60      │ 'drift_entry_quote_of_perp_trade'            │ 151              │
│ 61      │ 'solana_get_owned_domains'                   │ 150              │
│ 62      │ 'deposit_to_drift_user_account'              │ 149              │
│ 63      │ 'deposit_to_2by2_multisig'                   │ 147              │
│ 64      │ 'solana_openbook_create_market'              │ 146              │
│ 65      │ 'solana_allora_get_price_inference'          │ 145              │
│ 66      │ 'solana_restake'                             │ 144              │
│ 67      │ 'get_helius_webhook'                         │ 144              │
│ 68      │ 'delete_helius_webhook'                      │ 143              │
│ 69      │ 'create_drift_user_account'                  │ 142              │
│ 70      │ 'solana_stake'                               │ 140              │
│ 71      │ 'solana_tiplink'                             │ 140              │
│ 72      │ 'solana_token_data'                          │ 139              │
│ 73      │ 'solana_register_domain'                     │ 138              │
│ 74      │ 'solana_parse_transaction_helius'            │ 138              │
│ 75      │ 'solana_list_nft_for_sale'                   │ 137              │
│ 76      │ 'update_drift_vault_delegate'                │ 137              │
│ 77      │ 'solana_voltr_deposit_strategy'              │ 137              │
│ 78      │ 'solana_voltr_withdraw_strategy'             │ 137              │
│ 79      │ 'solana_lend_asset'                          │ 136              │
│ 80      │ 'request_withdrawal_from_drift_vault'        │ 135              │
│ 81      │ 'drift_perp_market_funding_rate'             │ 135              │
│ 82      │ 'deposit_into_drift_vault'                   │ 134              │
│ 83      │ 'request_unstake_from_drift_insurance_fund'  │ 134              │
│ 84      │ 'stake_to_drift_insurance_fund'              │ 131              │
│ 85      │ 'derive_drift_vault_address'                 │ 130              │
│ 86      │ 'solana_get_owned_tld_domains'               │ 125              │
│ 87      │ 'solana_cancel_nft_listing'                  │ 125              │
│ 88      │ 'drift_vault_info'                           │ 125              │
│ 89      │ 'solana_allora_get_inference_by_topic_id'    │ 125              │
│ 90      │ 'withdraw_from_drift_vault'                  │ 124              │
│ 91      │ 'unstake_from_drift_insurance_fund'          │ 124              │
│ 92      │ 'drift_lend_and_borrow_apy'                  │ 124              │
│ 93      │ 'solana_token_data_by_ticker'                │ 123              │
│ 94      │ 'orca_fetch_positions'                       │ 123              │
│ 95      │ 'solana_pyth_fetch_price'                    │ 123              │
│ 96      │ 'solana_manifest_create_market'              │ 122              │
│ 97      │ 'solana_voltr_get_position_values'           │ 122              │
│ 98      │ 'solana_get_info'                            │ 119              │
│ 99      │ 'does_user_have_drift_account'               │ 115              │
│ 100     │ 'solana_create_image'                        │ 114              │
│ 101     │ 'drift_user_account_info'                    │ 113              │
│ 102     │ 'solana_get_all_tlds'                        │ 110              │
│ 103     │ 'solana_request_funds'                       │ 106              │
│ 104     │ 'solana_allora_get_all_topics'               │ 106              │
│ 105     │ 'close_empty_token_accounts'                 │ 103              │
│ 106     │ 'solana_get_tps'                             │ 102              │
│ 107     │ 'solana_get_wallet_address'                  │ 100              │
└─────────┴──────────────────────────────────────────────┴──────────────────┘

Category Totals:
┌─────────┬─────────────────┬─────────────┐
│ (index) │ Langchain Total │ Grand Total │
├─────────┼─────────────────┼─────────────┤
│ 0       │ 18993           │ 18993       │
└─────────┴─────────────────┴─────────────┘

Implementation Details

uses gpt-4o tokenizer from tiktoken to count tokens

Transaction executed by agent

Example transaction:

Prompt Used

who owns toly.sol

Additional Notes

Future PRs should use this information to group similar tools together to minimize unused tokens.

Checklist

I have tested these changes locally
I have added the prompt used to test it

michaelessiet · 2025-02-02T15:50:00Z

LGTM @thearyanag

ngundotra added 2 commits February 1, 2025 21:02

setup token analysis for all functions

08db584

add pnpm tool-summary:langchain

2f0528e

michaelessiet approved these changes Feb 2, 2025

View reviewed changes

thearyanag merged commit 3807277 into sendaifun:main Feb 3, 2025
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup token analysis for all functions #256

Setup token analysis for all functions #256

ngundotra commented Feb 1, 2025 •

edited

Loading

michaelessiet commented Feb 2, 2025 •

edited

Loading

Setup token analysis for all functions #256

Setup token analysis for all functions #256

Conversation

ngundotra commented Feb 1, 2025 • edited Loading

Add Token Counting Utility

Changes Made

Implementation Details

Transaction executed by agent

Prompt Used

Additional Notes

Checklist

michaelessiet commented Feb 2, 2025 • edited Loading

ngundotra commented Feb 1, 2025 •

edited

Loading

michaelessiet commented Feb 2, 2025 •

edited

Loading