Ollama llama for your LLM

Using Ollama as a Practical Local LLM and How to Tune It Better

I have been looking at Ollama because it gives you a way to run LLMs locally without having to depend on cloud APIs for every test.

For me, the value is simple. You can build, test, and experiment with AI on your own machine. You are not paying for every prompt while you are still trying to figure things out. You also have more control over the data because the model is running locally.

Ollama is not going to replace every high-end commercial LLM. That is not the point. The point is that it is a very good practical tool for developers, consultants, and small teams who want to build AI solutions without a lot of overhead.

What Ollama Does

Ollama lets you run open-source models locally. You install it, pull a model, and then you can call it from the command line or through a local API.

Example:

ollama run llama3.2

You can also try other models:

ollama run mistral
ollama run qwen2.5
ollama run deepseek-r1

Once it is running, your application can call it locally:

http://localhost:11434

That means you can connect it to Python, Streamlit, Reflex, LangChain, or any custom app you are building.

Why I Like This Approach

The biggest reason is control.

When you are building AI tools, you test a lot. You change prompts. You test different inputs. You break things. You fix things. If every test is going through a paid API, the cost adds up.

With Ollama, I can test locally first. Once the idea works, then I can decide if it needs a cloud model or if the local model is enough.

This makes it useful for things like:

Document summaries, Internal assistants, SAP data explanations, Code generation, Testing prompts, RAG prototypes, Log analysis, JSON extraction, Small business AI tools

It gives you a working AI engine without needing a big setup.

Basic Setup

The setup is pretty simple.

On Linux you can install it like this:

curl -fsSL https://ollama.com/install.sh | sh

Then run a model:

ollama run llama3.2

From Python, you can call Ollama like this:

import requests

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2",
        "prompt": "Explain SAP archiving in simple business terms.",
        "stream": False
    }
)

print(response.json()["response"])

That is enough to start building a local assistant.

Do Not Start With the Biggest Model

One mistake is thinking bigger is always better.

If your machine cannot handle the model, it will be slow and frustrating. I would rather use a smaller model that runs fast than a huge model that barely works.

Start with something practical first.

For normal use:

ollama run llama3.2
ollama run mistral
ollama run qwen2.5

Then test which one gives you the best balance of speed and answer quality.

For coding, use a coding model.

For document search, use embeddings.

For business summaries, use a general instruction model.

The model should match the job.

Tuning Ollama for Better Performance

Ollama works right away, but tuning it makes a big difference.

The main things I look at are:

Context size,Temperature Output length, Repeat penalty, Model selection, Prompt format RAG design ,How long the model stays loaded

These small changes can make the local model a lot more useful.

Keep the Model Loaded

If the model has to load every time, it slows everything down.

Ollama has a keep_alive option. This keeps the model in memory after the request.

Example:

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2",
        "prompt": "Summarize this document.",
        "stream": False,
        "keep_alive": "30m"
    }
)

For a local app that you are using all day, keeping the model warm helps a lot.

If your machine has low memory, then use a shorter time.

Tune the Context

Context size controls how much text the model can look at.

In Ollama this is usually num_ctx.

Example:

"options": {
    "num_ctx": 4096
}

My simple rule:

2048  = faster
4096  = good starting point
8192+ = useful for larger documents, but heavier

Do not just make it huge because you can. Bigger context takes more memory and may slow things down.

If it is just a normal question, keep it smaller.

If you are reviewing documents, increase it only when needed.

Lower the Temperature for Business Work

Temperature controls how random the answer is.

For business work, I do not want the model being too creative. I want it to stay focused.

For SAP, reports, summaries, code, or data work, I would start here:

"options": {
    "temperature": 0.2
}

For JSON extraction, I would go lower:

"options": {
    "temperature": 0
}

For creative writing, then you can raise it:

"options": {
    "temperature": 0.7
}

For business use, lower is usually better.

Limit the Output

Sometimes the model keeps writing when you only need a short answer.

Use num_predict to control how much it generates.

Example:

"options": {
    "num_predict": 500
}

A simple guide:

150  = short answer
500  = normal answer
800  = technical answer
1200 = longer report

This helps with speed and keeps the answer from going all over the place.

Use Repeat Penalty

Local models can repeat themselves sometimes.

If that happens, use repeat_penalty.

Example:

"options": {
    "repeat_penalty": 1.1
}

If it still repeats, try:

"repeat_penalty": 1.2

Do not overdo it. Too much can make the answer sound strange.

Use a Modelfile

A Modelfile is useful because you can set the behavior once and reuse it.

Example:

FROM llama3.2

SYSTEM """
You are a practical business assistant.
Answer clearly.
Do not make things up.
If information is missing, say it is missing.
"""

PARAMETER temperature 0.2
PARAMETER num_ctx 4096
PARAMETER repeat_penalty 1.1

Create it:

ollama create business-assistant -f Modelfile

Run it:

ollama run business-assistant

Now you have your own tuned local assistant.

This is better than passing the same settings every time.

Use RAG Instead of Dumping Everything Into the Prompt

Do not paste a giant document into the prompt every time.

That is not a good design.

A better way is RAG.

The idea is simple:

Take the documents
Break them into chunks
Create embeddings
Store them in a vector database
Search for the most relevant chunks
Send only those chunks to Ollama
Let the model answer from that

This is much better for document work.

A simple stack could be:

Ollama for the answer Embedding model for search FAISS, Chroma, SQLite, or DuckDB for storage Streamlit or Reflex for the UI

This is enough to build a real local document assistant.

Use JSON Output When You Need Data

Ollama can also return structured output like JSON.

That is important for business automation.

Example:

import requests

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2",
        "prompt": "Extract vendor name, invoice number, date, and amount from this text: ...",
        "format": "json",
        "stream": False,
        "options": {
            "temperature": 0
        }
    }
)

print(response.json()["response"])

This is useful for:

Invoices Logs SAP metadata Archive object classification Document tagging Compliance checksReport generation

For me, this is where local LLMs become more than just chat.

Match the Model to the Task

I would not use one model for everything.

A better setup is:

Small model       = classification
Medium model      = summaries
Coding model      = Python or SQL
Embedding model   = document search
Reasoning model   = harder analysis

This gives better performance because each model is doing the job it is good at.

You do not need a large model just to classify a document.

But for code or deeper explanations, you may want a stronger one.

Simple Local Assistant Example

Here is a simple example:

import requests

def ask_ollama(question):
    payload = {
        "model": "business-assistant",
        "prompt": question,
        "stream": False,
        "keep_alive": "30m",
        "options": {
            "temperature": 0.2,
            "num_ctx": 4096,
            "num_predict": 600,
            "repeat_penalty": 1.1
        }
    }

    response = requests.post(
        "http://localhost:11434/api/generate",
        json=payload
    )

    response.raise_for_status()
    return response.json()["response"]

answer = ask_ollama("Explain the business value of SAP archiving in simple terms.")
print(answer)

This can be the start of a Streamlit app, Reflex app, document assistant, SAP helper, or internal business tool.

Settings I Would Start With

For business Q&A:

temperature: 0.2
num_ctx: 4096
num_predict: 500
repeat_penalty: 1.1
keep_alive: 30m

For coding:

temperature: 0.1
num_ctx: 4096
num_predict: 800
repeat_penalty: 1.05

For JSON extraction:

temperature: 0
num_ctx: 4096
num_predict: 500
format: json

For creative writing:

temperature: 0.7
num_ctx: 4096
num_predict: 1000
repeat_penalty: 1.1

These are not perfect for every case, but they are good starting points.

The Hardware Side

Ollama can run on a laptop, but hardware matters.

More RAM helps.A GPU helps.Smaller models are faster.Large context uses more memory.Quantized models are easier to run.

So do not fight your machine.

Start with a model that runs well. Then improve the workflow around it.

A smaller model with good prompts and RAG can be more useful than a big model that runs too slow.

Where Ollama Fits

I see Ollama fitting well for:

Local development AI prototypes Private document review Internal assistants Prompt testing Small business tools Coding helpData classification SAP and archive data experiments

It may not be the final answer for every enterprise production system. But it is a very strong tool for building and proving ideas fast.

Final Thought

Ollama is a practical local LLM tool.

The value is not just that it runs locally. The value is that you can build with it, test with it, and control it.

The key is to tune it the right way.

Pick the right model.Keep the model loaded.Control the context.Lower the temperature for business work.Limit the output.Use RAG for documents.Use JSON when you need structured data.

That is how you turn Ollama into a useful local AI engine instead of just another chatbot.