Skip to content

latitudegames/GPT-3-Encoder

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

4054d73 · Dec 25, 2022

History

40 Commits
Sep 8, 2020
Sep 8, 2020
Apr 21, 2021
Sep 8, 2020
Sep 4, 2020
Oct 31, 2020
Sep 4, 2020
Apr 21, 2021
Sep 4, 2020
Dec 25, 2022
Oct 31, 2020
Sep 8, 2020
Apr 21, 2021
Oct 31, 2020
Sep 4, 2020

Repository files navigation

GPT-3-Encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3

About

GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a javascript implementation of OpenAI's original python encoder/decoder which can be found here

Install with npm

npm install gpt-3-encoder

Usage

Compatible with Node >= 12

const {encode, decode} = require('gpt-3-encoder')

const str = 'This is an example sentence to try encoding out on!'
const encoded = encode(str)
console.log('Encoded this string looks like: ', encoded)

console.log('We can look at each token and what it represents')
for(let token of encoded){
  console.log({token, string: decode([token])})
}

const decoded = decode(encoded)
console.log('We can decode it back into:\n', decoded)

About

Javascript BPE Encoder Decoder for GPT-2 / GPT-3

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published