litellm/docs/extras/integrations/llms/ctransformers.ipynb

C Transformers

The C Transformers library provides Python bindings for GGML models.

This example goes over how to use LangChain to interact with C Transformers models.

Install

In [ ]:

%pip install ctransformers

Load Model

In [ ]:

from langchain.llms import CTransformers

llm = CTransformers(model="marella/gpt-2-ggml")

Generate Text

In [ ]:

print(llm("AI is going to"))

Streaming

In [ ]:

from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

llm = CTransformers(
    model="marella/gpt-2-ggml", callbacks=[StreamingStdOutCallbackHandler()]
)

response = llm("AI is going to")

LLMChain

In [ ]:

from langchain import PromptTemplate, LLMChain

template = """Question: {question}

Answer:"""

prompt = PromptTemplate(template=template, input_variables=["question"])

llm_chain = LLMChain(prompt=prompt, llm=llm)

response = llm_chain.run("What is AI?")