i can’t let llama even 65b write python code… #328

FNsi · 2023-03-20T14:25:37Z

FNsi
Mar 20, 2023

😅

FNsi · 2023-03-20T14:31:08Z

FNsi
Mar 20, 2023
Author

In quantisation 4_0

llama_model_load: loading model from 'models/65B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 8192
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 64
llama_model_load: n_layer = 80
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 22016
llama_model_load: n_parts = 8
llama_model_load: ggml ctx size = 41477.73 MB
llama_model_load: memory_size =  2560.00 MB, n_mem = 40960
llama_model_load: loading model part 1/8 from 'models/65B/ggml-model-q4_0.bin'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 2/8 from 'models/65B/ggml-model-q4_0.bin.1'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 3/8 from 'models/65B/ggml-model-q4_0.bin.2'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 4/8 from 'models/65B/ggml-model-q4_0.bin.3'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 5/8 from 'models/65B/ggml-model-q4_0.bin.4'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 6/8 from 'models/65B/ggml-model-q4_0.bin.5'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 7/8 from 'models/65B/ggml-model-q4_0.bin.6'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 8/8 from 'models/65B/ggml-model-q4_0.bin.7'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723

system_info: n_threads = 16 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 

main: prompt: ' write a python program that prints the first 10 fibonacci numbers.'
main: number of tokens in prompt = 17
     1 -> ''
  2436 -> ' write'
   263 -> ' a'
  3017 -> ' python'
  1824 -> ' program'
   393 -> ' that'
 14677 -> ' prints'
   278 -> ' the'
   937 -> ' first'
 29871 -> ' '
 29896 -> '1'
 29900 -> '0'
 18755 -> ' fib'
   265 -> 'on'
 21566 -> 'acci'
  3694 -> ' numbers'
 29889 -> '.'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000


 write a python program that prints the first 10 fibonacci numbers.
 # Write code to print first n Fibonaci number in Python ? [end of text]


main: mem per token = 71159620 bytes
main:     load time = 22742.83 ms
main:   sample time =     8.23 ms
main:  predict time = 32789.13 ms / 993.61 ms per token

apx run ./main -m models/30B/ggml-model-q4_0.bin  -p "write a python program that prints the first 10 fibonacci numbers."
main: seed = 1679322313
llama_model_load: loading model from 'models/30B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 6656
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 52
llama_model_load: n_layer = 60
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 17920
llama_model_load: n_parts = 4
llama_model_load: ggml ctx size = 20951.50 MB
llama_model_load: memory_size =  1560.00 MB, n_mem = 30720
llama_model_load: loading model part 1/4 from 'models/30B/ggml-model-q4_0.bin'
llama_model_load: ................................................................... done
llama_model_load: model size =  4850.14 MB / num tensors = 543
llama_model_load: loading model part 2/4 from 'models/30B/ggml-model-q4_0.bin.1'
llama_model_load: ................................................................... done
llama_model_load: model size =  4850.14 MB / num tensors = 543
llama_model_load: loading model part 3/4 from 'models/30B/ggml-model-q4_0.bin.2'
llama_model_load: ................................................................... done
llama_model_load: model size =  4850.14 MB / num tensors = 543
llama_model_load: loading model part 4/4 from 'models/30B/ggml-model-q4_0.bin.3'
llama_model_load: ................................................................... done
llama_model_load: model size =  4850.14 MB / num tensors = 543

system_info: n_threads = 16 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 

main: prompt: ' write a python program that prints the first 10 fibonacci numbers.'
main: number of tokens in prompt = 17
     1 -> ''
  2436 -> ' write'
   263 -> ' a'
  3017 -> ' python'
  1824 -> ' program'
   393 -> ' that'
 14677 -> ' prints'
   278 -> ' the'
   937 -> ' first'
 29871 -> ' '
 29896 -> '1'
 29900 -> '0'
 18755 -> ' fib'
   265 -> 'on'
 21566 -> 'acci'
  3694 -> ' numbers'
 29889 -> '.'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000


 write a python program that prints the first 10 fibonacci numbers.
use list comprehension to print only odd fiboanrci numbers less than or equal to n where n =35 . [end of text]


main: mem per token = 43600900 bytes
main:     load time = 11926.69 ms
main:   sample time =    13.68 ms
main:  predict time = 22053.11 ms / 501.21 ms per token

2 replies

bitRAKE Mar 21, 2023

Try a prompt like:

The following Python function will tokenize the text:
\begin{code}
def tokenize(text, keywords, delimiters):

The trick is to imagine what you want and then give the model the start of that. In general it isn't conversational - like ChatGPT. It's like lossy compressed text.

FNsi Mar 21, 2023
Author

!gonna try as soon as I can!

FNsi · 2023-03-20T14:33:12Z

FNsi
Mar 20, 2023
Author

Quantisation 4_1 30b

apx run ./main -m models/30B/ggml-model-q4_1.bin  -p "write a python program that prints the first 10 fibonacci numbers."
main: seed = 1679322416
llama_model_load: loading model from 'models/30B/ggml-model-q4_1.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 6656
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 52
llama_model_load: n_layer = 60
llama_model_load: n_rot   = 128
llama_model_load: f16     = 3
llama_model_load: n_ff    = 17920
llama_model_load: n_parts = 4
llama_model_load: ggml ctx size = 24829.16 MB
llama_model_load: memory_size =  1560.00 MB, n_mem = 30720
llama_model_load: loading model part 1/4 from 'models/30B/ggml-model-q4_1.bin'
llama_model_load: ................................................................... done
llama_model_load: model size =  5819.56 MB / num tensors = 543
llama_model_load: loading model part 2/4 from 'models/30B/ggml-model-q4_1.bin.1'
llama_model_load: ................................................................... done
llama_model_load: model size =  5819.56 MB / num tensors = 543
llama_model_load: loading model part 3/4 from 'models/30B/ggml-model-q4_1.bin.2'
llama_model_load: ................................................................... done
llama_model_load: model size =  5819.56 MB / num tensors = 543
llama_model_load: loading model part 4/4 from 'models/30B/ggml-model-q4_1.bin.3'
llama_model_load: ................................................................... done
llama_model_load: model size =  5819.56 MB / num tensors = 543

system_info: n_threads = 16 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 

main: prompt: ' write a python program that prints the first 10 fibonacci numbers.'
main: number of tokens in prompt = 17
     1 -> ''
  2436 -> ' write'
   263 -> ' a'
  3017 -> ' python'
  1824 -> ' program'
   393 -> ' that'
 14677 -> ' prints'
   278 -> ' the'
   937 -> ' first'
 29871 -> ' '
 29896 -> '1'
 29900 -> '0'
 18755 -> ' fib'
   265 -> 'on'
 21566 -> 'acci'
  3694 -> ' numbers'
 29889 -> '.'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000


 write a python program that prints the first 10 fibonacci numbers.
The Fibonacci sequence is defined in terms of itself, or by recursion. The initial values are 1 and 2. Each successive value (starting from second) equals the sum of its two previous elements:
(n-3)+((n)-4)= (n+5/6 ) +(-9/8)*( n+7/8) where n is a fibonacci number, and x^y means ‘x to y power’. The Fibonacci sequence can be expressed in terms of the golden ratio as well:
The sum of any two adjacent numbers (except first) equals the next one – it follows that each successive pair sums up 1 less than its neighboring number, and so on. Let n = #th fibonacci term, then f(n+2)=fib + 0; F(n-3)+F(n-4). It’s a bit like the way you add pairs of numbers when multiplying binomial terms together!
Here is my solution in python: [end of text]


main: mem per token = 43600900 bytes
main:     load time = 14208.50 ms
main:   sample time =   110.65 ms
main:  predict time = 168038.94 ms / 709.03 ms per token

1 reply

FNsi Mar 20, 2023
Author

Tbh it's a good answer anyway...

Green-Sky · 2023-03-20T14:35:33Z

Green-Sky
Mar 20, 2023
Collaborator

It seems you are treating the raw model as if, it was instruction tuned. In it's raw form you need to think of a text predictor.

you can start with a code comment describing the functionality and supplying the function signature/declaration.

1 reply

FNsi Mar 20, 2023
Author

😑nahhh, sad story... meanwhile 30b quantisation 4.1 seems obvious better than others.

tarruda · 2023-03-20T15:18:45Z

tarruda
Mar 20, 2023

@FNsi you might find this useful: https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/

1 reply

FNsi Mar 21, 2023
Author

👍

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

i can’t let llama even 65b write python code… #328

{{title}}

Replies: 4 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

i can’t let llama even 65b write python code… #328

FNsi Mar 20, 2023

Replies: 4 comments · 5 replies

FNsi Mar 20, 2023 Author

bitRAKE Mar 21, 2023

FNsi Mar 21, 2023 Author

FNsi Mar 20, 2023 Author

FNsi Mar 20, 2023 Author

Green-Sky Mar 20, 2023 Collaborator

FNsi Mar 20, 2023 Author

tarruda Mar 20, 2023

FNsi Mar 21, 2023 Author

FNsi
Mar 20, 2023

Replies: 4 comments 5 replies

FNsi
Mar 20, 2023
Author

FNsi Mar 21, 2023
Author

FNsi
Mar 20, 2023
Author

FNsi Mar 20, 2023
Author

Green-Sky
Mar 20, 2023
Collaborator

FNsi Mar 20, 2023
Author

tarruda
Mar 20, 2023

FNsi Mar 21, 2023
Author