notes

Log | Files | Refs

llama-cpp.txt (1076B)

      1 LLAMA-CPP
      2 
      3 Quick start:
      4 llama-cpp -m models/path/to/model.gguf -n 128 -p "Explain quantum physics"
      5 
      6 Optimizations:
      7 - Use -ngl 32 for GPU offloading (NVIDIA/AMD)
      8 - Use -rl 2048 for context window
      9 - Use --threads [number of physical cores]
     10 
     11 ====================================================================
     12 Commands:
     13 
     14 git clone git@github.com:ggml-org/llama.cpp.git
     15 
     16 cd llama.cpp
     17 
     18 Build with CUDA:
     19 
     20 cmake -B build -DGGML_CUDA=ON
     21 cmake --build build --config Release
     22 
     23 Build CPU:
     24 
     25 cmake -B build
     26 cmake --build build --config Release
     27 
     28 Run on NVIDIA GPU without reasoning:
     29 
     30 ./build/bin/llama-server -hf unsloth/gemma-4-12b-it-GGUF -ngl 999 -c 32768 --host 0.0.0.0 --port 9090 --reasoning off --parallel 1
     31 
     32 Run on CPU only without reasoning:
     33 
     34 ./build/bin/llama-server -hf unsloth/gemma-4-12b-it-GGUF -c 32768 --host 0.0.0.0 --port 9090 --reasoning off --parallel 1
     35 
     36 ====================================================================
     37 Check how many requests it can accept in parallel:
     38 
     39 curl -s 127.0.0.1:9090/props | jq '{n_ctx: .default_generation_settings.n_ctx, total_slots}'