Max seq length

[baochi0212]

System Info

Thanks team for great work,
As I see in the report the max ctx length is 16k right ? But the config max pos embed is 4k, and when I modify to 8k the result and memory just work within 4k. Am I missing sth ?, thanks.

  "num_mem_blocks": 2,
  "max_position_embeddings": 4096,
  "lora_rank": 128,
  "mamba_headdim": 64,
  "use_mem_rope": false,
  "rope_theta": 10000,
  "model_type": "zamba2",
  "num_attention_heads": 32,
  "num_hidden_layers": 54,
  "num_key_value_heads": 32,
  "num_logits_to_keep": 1,
  "num_query_groups": 32,
  "pad_token_id": 0,
  "rms_norm_eps": 1e-05,
  "sliding_window": null,
  "state_size": 64,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.43.0.dev0",
  "use_cache": true,
  "use_mamba_kernels": true,
  "use_shared_block_lora": true,
  "se_shared_attention_lora": false,
  "vocab_size": 32000
}

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

example code

Expected behavior

16k context length

[Yarflam]

They added the parameter use_long_context=True (see HF page) but it's not working better:

I have erroneous generations when I generate an answer from a long context :

Thank you for your future support! Zamba2 looks like really good. :)