How does llama.cpp use UNK token in generation?

[m1chae1bx]

Hello, I wanted to know how does the UNK token is used in generation, or if it is even used. I assume the UNK token is used when there is no token with high probability during the decoding process. Does anyone have an idea?

[cmp-nct]

Did you ever see it used ?

Mistral, llama2, Falcon they all use BPE tokenization so they are not really short of expression.
UNK is supposed to be used for unknown words that can not be tokenized, with BPE you can tokenize everything and if something can not be tokenized llama.cpp currently crashes :) So no UNK there.

I don't know for certain but my guess is that UNK is mostly a relict of older smaller language models.

[crasm]

The only time I've seen UNK is when I messed up my code calling into llama.cpp and got the order of things wrong, or didn't fill out the batch struct properly. UNK has the token ID of 0.

[ponikar]

hey @crasm I am facing the same issue right now, I am trying to make it work in my react native iOS project, model is being loaded correctly but the output is unk unk unk. I tried things such as token sampling but it did not work.

Here's my code, it would be great if anyone can help me out with this. I have no background of ML/AI. I was experimenting with llama.cpp bindings.

#import "LocalModel.h"
#include "llama.h"
#include <regex>
#include <vector>
#include "common.h"
#include <string>

using namespace std;

@implementation LocalModel {
    llama_context* ctx;
    llama_model* model;
}

RCT_EXPORT_MODULE()

- (instancetype)init {
    self = [super init];
    if (self) {
        llama_backend_init();
        NSLog(@"LocalModel: Backend initialized");
    }
    return self;
}

RCT_EXPORT_METHOD(loadModelAndAskQuestion:(NSString *)modelName
                  question:(NSString *)question
                  resolve:(RCTPromiseResolveBlock)resolve
                  reject:(RCTPromiseRejectBlock)reject) {
    NSString *modelPath = [[NSBundle mainBundle] pathForResource:@"tinyllama-1.1b-chat-v1.0.Q2_K" ofType:@"gguf"];
    NSLog(@"LocalModel: Attempting to load model from path: %@", modelPath);
  
    llama_model* model;
    llama_context* ctx;
    llama_model_params model_params;
    llama_context_params context_params;
    
    model_params = llama_model_default_params();
    context_params = llama_context_default_params();

            // Set default parameters
    context_params.n_ctx = 2048;
    context_params.n_threads = 4;
    context_params.n_threads_batch = 4;
    
    model = llama_load_model_from_file([modelPath UTF8String], model_params);
    //    resolve([]);
    
    ctx = llama_new_context_with_model(model, context_params);
    
    
    std::string prompt = "Tell me more about elon musk and spaceX?";
    std::vector<llama_token> tokens = llama_tokenize(ctx, prompt, false);
    llama_batch batch = llama_batch_init(tokens.size(), 0, 1);
    
    for (size_t i = 0; i < tokens.size(); i++) {
        llama_batch_add(batch, tokens[i], i, { 0 }, false);
    }
    
    if (llama_decode(ctx, batch) != 0) {
                throw std::runtime_error("Failed to decode");
    }
    
    int max_tokens = 100;
    std::string result = prompt;
            for (int i = 0; i < max_tokens; i++) {
                llama_token new_token_id = 0;
                auto logits = llama_get_logits(ctx);
                auto n_vocab = llama_n_vocab(model);

                std::vector<llama_token_data> candidates;
                candidates.reserve(n_vocab);
                for (llama_token token_id = 0; token_id < n_vocab; token_id++) {
                    candidates.emplace_back(llama_token_data{ token_id, logits[token_id], 0.0f });
                }

                llama_token_data_array candidates_p = { candidates.data(), candidates.size(), false };
                new_token_id = llama_sample_token_greedy(ctx, &candidates_p);

                if (new_token_id == llama_token_eos(model)) {
                    break;
                }
                
                
                NSLog(@"TOKEN ->>>>>> %s", llama_token_to_piece(ctx, new_token_id).c_str());

                result += llama_token_to_piece(ctx, new_token_id);

                batch = llama_batch_init(1, 0, 1);
                llama_batch_add(batch, new_token_id, batch.n_tokens, { 0 }, false);
                if (llama_decode(ctx, batch) != 0) {
                    throw std::runtime_error("Failed to decode");
                }
            }
    
    
}

- (void)dealloc {
    llama_backend_free();
    NSLog(@"LocalModel: Backend freed");
}

@end