llama-cpp-python - Harness the power of llama.cpp with seamless Python integration.

llama-cpp-python

9 views

Harness the power of llama.cpp with seamless Python integration.

Pitch

Discover llama-cpp-python, the easy way to leverage the capabilities of the llama.cpp library through Python. With a simple, intuitive API, you can access text completion, model compatibility, and even multimodal functionalities. Elevate your coding experience with powerful tools that enhance development without the complexity.

Description

This repository provides simple Python bindings for llama.cpp, a powerful C++ library designed for text generation and language modeling. With llama-cpp-python, developers can easily harness the capabilities of llama.cpp directly within their Python applications.

Key Features:

Low-Level C API Access: Utilize the native C API through the ctypes interface for advanced performance and flexibility.
High-Level Python API: Designed for ease of use, it allows for text completion with an OpenAI-like API, ensuring seamless integration for those familiar with OpenAI's offerings.
LangChain and LlamaIndex Compatibility: Easily integrate with popular frameworks such as LangChain and LlamaIndex for enhanced functionality.
OpenAI Compatible Web Server: Run a local web server that mimics the OpenAI API, facilitating easy deployment and testing. Key features include:
- Local Copilot Replacement: Access code completion functionality similar to OpenAI's Copilot.
- Function Calling Support: Invoke functions directly from your prompts, enhancing interactivity.
- Vision API: Utilize multi-modal models that support processing both text and images.
- Multiple Models: Easily switch between various models to suit your project needs.

Example Usage:

Here’s a quick example showcasing how to use the high-level API for basic text completion:

from llama_cpp import Llama

llm = Llama(model_path="./models/7B/llama-model.gguf")
output = llm(
    "Q: Name the planets in the solar system? A: ",
    max_tokens=32,
    stop=["Q:", "\n"],
    echo=True
)
print(output)

The llama-cpp-python package makes it straightforward to generate text completions, and its detailed documentation is available at Read the Docs.

Development and Contributions:

This project is actively maintained, and contributions are welcome. To contribute, clone the repository and install it in an editable mode:

git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python.git
cd llama-cpp-python
pip install -e .

Overall, llama-cpp-python empowers developers to leverage cutting-edge language models, making it a robust choice for text generation tasks.