The head of customer service at an online retailer has a budget for a chatbot and three demos on the table. All three look impressive: the bot answers questions about returns, delivery and payment in fluent language, in any phrasing. Six weeks after launch, the numbers tell a different story. Contacts to the call centre have not fallen, because customers who got a confident but wrong answer about a return deadline call anyway — now irritated. One conversation, in which the bot promised a refund the policy did not allow, has reached social media.
The model in that project was not the problem. The problem was everything around it: which documents the bot relied on, what it was allowed to promise, when it had to hand over to a person and how anyone would know whether it was answering correctly. That is where chatbot development actually happens in 2026 — and where most of the budget should go.
This guide is for a CTO or head of customer service deciding whether and how to build an AI chatbot. We cover the types of chatbots and when each makes sense, the architecture of a production assistant, how to choose a model, a costed example of building and running one, the regulatory duties that have applied in the EU since August 2026, the metrics that show whether a bot works and how to choose a development partner.
What does chatbot development mean in 2026?
“Chatbot” covers four quite different kinds of system, and confusing them is the most common source of mismatched expectations between a business and its vendor.
Rule-based chatbots follow decision trees: buttons, menus and predefined paths. They are predictable, cheap to run and easy to audit, but they handle only the scenarios someone explicitly designed. Any question outside the tree ends in “I don’t understand”.
Intent-based (NLU) chatbots recognise what the user wants — “track order”, “change address” — using a classifier trained on example phrases, then run a scripted flow for that intent. This was the dominant approach before large language models. It works well for a limited set of frequent requests, but each new intent requires training data and a new flow.
LLM chatbots with retrieval (RAG) understand free-form language and generate answers based on company knowledge retrieved at the moment of the question — policies, product data, help-centre articles. Retrieval-augmented generation is what makes the bot answer from your documents rather than from what the model remembers from training. This is the default architecture for customer-facing assistants in 2026.
Agentic chatbots with tools go a step further: besides answering, they act. They check an order status in the OMS, change a delivery slot, open a return, issue a voucher within set limits. Technically this means the model can call functions or APIs, increasingly through standards such as the Model Context Protocol. It is also where the risk profile changes most: a bot that can only talk can mislead; a bot that can act can also make costly mistakes.
Gartner predicted in March 2025 that by 2029 agentic AI will autonomously resolve 80% of common customer service issues, cutting operating costs by 30%. Whether or not that date holds, the direction is clear: the value of a chatbot increasingly comes from what it can do, not only what it can say.
Which type of chatbot does your use case need?
The right type depends on three questions: how varied the incoming questions are, whether answers can be taken from documents, and whether the bot needs to change anything in your systems.
| Use case | Recommended type | Why |
|---|---|---|
| A handful of fixed processes (booking, simple status check) | Rule-based or NLU | Predictable, cheap, easy to certify; an LLM adds risk without adding value |
| FAQ and policy questions phrased in many ways | LLM + RAG | Covers variations of wording; answers grounded in current documents |
| Questions requiring customer data (order, invoice, contract) | LLM + RAG + read-only tools | Needs authenticated access to data, but no write operations |
| Actions: returns, rescheduling, cancellations, small compensations | Agentic with tools and limits | Real automation of work; requires authorisation rules, limits and audit |
| Internal assistant for employees (HR, IT helpdesk, sales support) | LLM + RAG, sometimes agentic | Lower reputational risk, higher tolerance for imperfect answers |
| Regulated advice (credit, insurance, health) | Hybrid: scripted core, LLM for language only | Decisions must follow rules; the model should explain, not decide |
Two practical observations from projects. First, most companies do not need one type — they need a combination. A well-designed assistant uses deterministic flows for the few processes where a mistake is expensive and an LLM for the long tail of questions. Second, it pays to start with read-only access. A bot that can look up an order but not modify it delivers most of the value in the first months, while the team learns how customers actually talk to it.
What does the architecture of a production AI chatbot look like?
A demo chatbot is a prompt, a model and a chat window. A production assistant has at least eight components, and most of the engineering effort goes into the ones that are invisible in a demo.
- Channels. Web widget, mobile app, WhatsApp, Messenger, email, voice. Each channel has its own constraints on message length, formatting and authentication.
- Orchestration layer. The logic that decides what happens with each message: classify it, choose a deterministic flow or the LLM path, call tools, apply guardrails and decide on escalation.
- The model. One or more large language models, often a cheaper model for classification and routing and a stronger one for complex answers.
- Knowledge and retrieval. Ingestion of documents, splitting into chunks, embeddings, a vector or hybrid search index, and — the part most often neglected — a process for keeping the content current. A bot answering from last year’s returns policy is worse than no bot.
- Tools and integrations. APIs of the OMS, CRM, billing or ticketing system, with authentication, permissions and limits on what the bot may do.
- Guardrails. Input filtering (prompt injection, abuse, out-of-scope requests), output checks (answers without a source, forbidden commitments, personal data leaks) and topic boundaries.
- Human handoff. Escalation to an agent with the full conversation history and a summary, so the customer does not have to repeat everything.
- Observability and evaluation. Logging of conversations, retrieved sources and tool calls; automated evaluation on a test set; dashboards for quality, cost and escalations.
Components 4, 6, 7 and 8 decide whether the chatbot is trusted. We covered the engineering side in more depth in our LLM integration checklist for enterprise applications; here it is enough to say that if a proposal from a vendor describes the model in detail but devotes one line to knowledge maintenance and evaluation, the proposal is incomplete.
How do you choose the model: a commercial API or an open-weight model?
The choice of model matters less than it did two years ago, because the gap between leading models on typical customer service tasks has narrowed and prices have fallen sharply. It still has three consequences: cost, data control and operational effort.
Commercial APIs (OpenAI, Anthropic, Google and others) offer the strongest models with no infrastructure to run. Pricing is per million tokens, separately for input and output. According to price comparisons from September 2026, the market falls into three tiers: flagship models at roughly USD 4–10 per million input tokens and USD 20–50 per million output tokens, mid-tier models at around USD 2 input and USD 10 output, and budget models from about USD 0.10–0.30 input and USD 0.40–2.50 output. Prices change several times a year, so treat these as orders of magnitude and check the current price lists before budgeting.
Open-weight models run on your own infrastructure or a private cloud. They give full control over data and no per-token fees, but you pay for GPUs, operations and the team that keeps them running. For most companies with tens of thousands of conversations a month, the API is cheaper in total cost; self-hosting starts to make sense at very high volumes, with strict data residency requirements or where the model must run on premises.
Three practical rules apply regardless of vendor:
- Use more than one model. Routing simple questions and classification to a budget model and only complex answers to a stronger one typically cuts token costs substantially without a visible drop in quality.
- Use prompt caching. The system prompt and much of the retrieved context repeat across conversations; providers that support caching charge a fraction of the price for cached input.
- Keep the model swappable. An abstraction layer and your own evaluation set let you change the model when a better or cheaper one appears — which, at the current pace, happens every few months.
A broader comparison of AI tools for business is in our article on alternatives to ChatGPT.
How do multiple languages change a chatbot project?
Most material on chatbot development assumes English. A company serving customers in several countries — for example across the EU — faces a few issues that change both cost and quality.
Tokens cost more in some languages. General-purpose tokenizers split words in inflected languages such as Polish into more tokens than English words; dedicated tokenizers, like the Polish-optimised one introduced in version 3 of the open Bielik model, exist precisely to address this. With a commercial API, the same conversation in such a language consumes more tokens and leaves a shorter effective context window. Budget from measured usage on real conversations in each language, not from English examples.
Customers write carelessly. Chat messages are full of missing diacritics, typos, abbreviations and mixed languages. Keyword-only retrieval handles this poorly; semantic retrieval with multilingual embeddings handles it much better. The evaluation set should include such messages, not only neatly phrased questions written by the project team.
Inflection breaks simple rules. Deterministic guardrails that check for specific phrases in English must cope with inflected forms in other languages. Checks based on intent classification and on whether an answer contains a commitment confirmed by a tool work better than lists of forbidden words.
One truth in many languages. You must decide whether the bot answers from documents written separately in each language or from one source version plus translation. The first gives more natural answers but risks content drifting apart between countries; the second keeps facts consistent at some cost to fluency. In practice, one source version of policies plus local help articles works best, with automated checks that answers to the same question agree on the facts across languages.
Each language is its own test set. A model that performs very well in English may perform worse in Polish, Czech or Romanian. Report evaluation results separately per language and decide on launching each new country based on its own numbers.
How much does it cost to build and run an AI chatbot?
The honest answer depends on scope, so we calculated one concrete example with explicit assumptions. The project: an LLM + RAG customer service assistant for a retailer, with read-only access to order status, integration with the ticketing system for handoff, a web widget and an evaluation pipeline.
Build — 12 weeks. The team and rates follow our body leasing rate benchmark for 2026 for senior engineers; QA and analyst rates are our assumption. Amounts in Polish złoty, with US dollars at an assumed 3.70 PLN per USD.
| Role | Rate (PLN/h) | Allocation | Cost |
|---|---|---|---|
| Senior AI/ML engineer | 290 | 100% | PLN 139,200 |
| Senior backend developer | 220 | 100% | PLN 105,600 |
| QA engineer (mid) | 140 | 50% | PLN 33,600 |
| Business analyst / project manager | 200 | 30% | PLN 28,800 |
| Total build | PLN 307,200 (≈ USD 83,000) |
Run — per month. Assumptions: 20,000 conversations a month, six turns each, about 4,000 input tokens per turn (system prompt, retrieved context and history) and 300 output tokens, on a mid-tier model at USD 2 / USD 10 per million tokens.
| Item | Monthly cost |
|---|---|
| Model tokens (480M input, 36M output) | USD 1,320 ≈ PLN 4,884 |
| Infrastructure: vector index, hosting, logging (assumption) | USD 800 ≈ PLN 2,960 |
| Maintenance and evaluation: 0.25 FTE at PLN 250/h | PLN 10,500 |
| Total run | ≈ PLN 18,344 (≈ USD 5,000) |
That is about PLN 0.92 (USD 0.25) per conversation. Two things in this table are worth noticing. First, tokens are only about a quarter of the running cost; the largest item is people keeping the knowledge current and the quality measured. With prompt caching covering 60% of input, the token bill falls from USD 1,320 to about USD 800. Second, model choice moves the total less than intuition suggests: a budget model would cost about USD 230 a month in tokens, a flagship model about USD 6,600. The difference between them is real, but what decides the project’s economics is whether the bot resolves conversations — which is the next question. The path from prototype to production budget is covered in more detail in our guide to AI implementation cost.
When does a chatbot pay for itself?
A chatbot pays for itself through conversations it resolves that would otherwise need a person. To show the mechanics, assume a fully loaded cost of PLN 10 per chat handled by an agent (for example PLN 60 an hour and six chats an hour — replace it with your own figure).
| Share of conversations resolved by the bot | Agent cost avoided per month | Net saving per month | Payback of the build |
|---|---|---|---|
| 30% | PLN 60,000 | PLN 41,656 | 7.4 months |
| 50% | PLN 100,000 | PLN 81,656 | 3.8 months |
| 70% | PLN 140,000 | PLN 121,656 | 2.5 months |
On paper, the case is strong even at 30%. In practice, the word “resolved” carries all the weight. A conversation the bot closes with a wrong answer is not a saving: it comes back as a phone call, a complaint or a lost customer.
Klarna is the best-documented example of both sides of this. In February 2024 it announced that its AI assistant had handled 2.3 million conversations in its first month — two-thirds of customer service chats, the equivalent of 700 full-time agents — with satisfaction on par with humans. In May 2025 its CEO admitted that cost had been “a too predominant evaluation factor” and that the result was lower quality; the company began hiring human agents again and guaranteed customers access to a person. The lesson is not that chatbots do not work — Klarna still runs its assistant at scale — but that the business case must be built on resolution quality, not on volume.
The same applies to the real cost of AI: most of it is not the model, but keeping the data behind it correct.
How do you keep a chatbot from making things up?
LLMs generate plausible text; they do not know whether it is true. In customer service that is not a theoretical risk. In Moffatt v. Air Canada (2024), the airline’s chatbot told a customer he could apply for a bereavement fare retroactively, contradicting the company’s own policy page. British Columbia’s Civil Resolution Tribunal held Air Canada liable for negligent misrepresentation, awarded CAD 650.88 in damages plus interest and fees, and rejected the argument that the chatbot was a separate legal entity responsible for its own actions. The amount was small; the principle is not: the company answers for what its bot says.
Reducing that risk takes several layers, none of which is sufficient alone:
- Grounding. The bot answers only from retrieved sources and cites them. If retrieval finds nothing relevant, the correct answer is “I don’t know — let me connect you with someone”, not an improvisation.
- Knowledge ownership. Every source document has an owner and a review date. Most wrong answers we see in production come from outdated or contradictory documents, not from the model inventing things.
- Commitment guardrails. The bot must not promise refunds, discounts, deadlines or exceptions unless a tool confirms them. These are checked on output, with deterministic rules rather than instructions in the prompt alone.
- Scope limits. Questions outside the defined domain — legal advice, medical questions, competitors — are politely declined or escalated.
- Prompt injection defences. Users will try to make the bot ignore its instructions. Input filtering, separation of instructions from retrieved content and least-privilege tool permissions limit the damage when they succeed.
- Continuous evaluation. A test set of real questions with approved answers, run on every change of model, prompt or knowledge base, and a sample of live conversations reviewed by people every week.
Good prompt engineering helps, but a prompt is not a control. Anything that must never happen needs a check outside the model.
What does the EU AI Act require from chatbots since August 2026?
The EU Artificial Intelligence Act (Regulation (EU) 2024/1689) applies in stages. For customer-facing chatbots, the key date was 2 August 2026, when the transparency obligations in Article 50 became applicable. The Digital Omnibus package postponed obligations for high-risk AI systems to December 2027, but it did not postpone Article 50.
In practice this means:
- Disclosure. Providers must design AI systems that interact directly with people so that those people are informed they are dealing with AI, unless it is obvious from the context. For a chatbot, a clear notice at the start of the conversation is the standard solution.
- Marking generated content. Systems generating synthetic text, audio, images or video must mark outputs in a machine-readable way. Generative systems already on the market before 2 August 2026 received until 2 December 2026 for this marking requirement.
- Penalties. Breaches of these obligations can be fined up to EUR 15 million or 3% of global annual turnover, whichever is higher.
A chatbot becomes subject to far heavier obligations if it is used in a high-risk area listed in the Act — for example, evaluating creditworthiness or access to essential services. If your assistant makes or materially influences such decisions, the architecture has to be designed for the high-risk regime from the start, even with the later deadline.
GDPR applies in parallel. Chat logs are personal data, often including order details, addresses or complaints about health. You need a legal basis and a retention period for transcripts, a data processing agreement with the model provider, a check of where data is processed, and a decision on whether conversations may be used to improve the model — for most commercial APIs, business tiers do not train on customer data by default, but this must be confirmed contractually. For US companies serving EU customers, the same rules apply to those customers.
When should the chatbot hand over to a human?
Customer attitudes set the design constraint. In a Gartner survey of 5,728 customers, published in July 2024, 64% said they would prefer companies not to use AI in customer service, and the main concern was that it would become harder to reach a person. A chatbot that hides the human option does not just irritate — it confirms the fear that keeps customers away from the channel.
Good handoff design follows a few rules:
- The human option is always visible. Not as a reward for persistence, but as a normal part of the interface.
- Escalation triggers are explicit. Low retrieval confidence, repeated rephrasing by the customer, negative sentiment, specific topics (complaints, cancellations, vulnerable customers) and any request the bot is not authorised to fulfil.
- Context travels with the customer. The agent receives the transcript, a summary, the retrieved sources and the customer’s identity, so the conversation continues rather than restarts.
- Business hours are handled honestly. Outside agent hours the bot says when a person will respond and creates a ticket, instead of pretending to solve everything.
- Escalations are a learning source. Every handoff is labelled with a reason. The most frequent reasons tell you where to add knowledge, tools or rules next.
Handoff is not a failure metric. A bot that escalates the right 30% of conversations quickly, with full context, often produces better customer satisfaction and lower total cost than one that tries to keep 80% and gets a quarter of them wrong.
Which metrics show whether a chatbot works?
One number cannot describe a chatbot. The metrics below only make sense together, because each can be gamed on its own.
- Containment rate — the share of conversations that end without a human. Easy to measure and easy to inflate: a bot that makes escalation difficult scores high.
- Resolution rate — the share of conversations in which the customer’s issue was actually resolved, measured by confirmation, by the absence of a repeat contact within a set period (for example seven days) or by review of a sample.
- Repeat contact rate — how many customers come back through another channel with the same issue. This is the metric that exposes false containment.
- Deflection — the reduction in contacts to other channels (phone, email) after launch, compared with a baseline. It shows the business effect, but needs careful comparison with seasonality.
- CSAT after bot conversations — asked briefly at the end, compared with CSAT for human conversations on similar topics.
- Escalation quality — how long the agent needs after a handoff and whether the summary was sufficient.
- Answer accuracy on the test set — the share of correct, grounded answers in automated evaluation, tracked for every release.
- Cost per resolved conversation — total running cost divided by resolved conversations, not by all conversations.
For an internal view, add latency, tool-call error rate and the share of answers without a source. For the board, three numbers usually suffice: resolution rate, cost per resolved conversation and CSAT compared with human service.
What does the chatbot development process look like step by step?
The projects that go well follow a similar sequence, and the steps that are skipped most often are the ones that decide success.
- Discovery (2–3 weeks). Analyse real conversations from the last few months: the top intents, their share of volume, which ones need data or actions, and which are risky. Define what the bot will not do. Set target metrics.
- Knowledge audit. Inventory the documents the bot will rely on, assign owners, remove contradictions and outdated versions. In our experience this step often takes longer than building the retrieval pipeline.
- Evaluation set. Before writing prompts, collect a few hundred real questions with approved answers and expected behaviour, including questions the bot should refuse or escalate.
- Prototype. Build the core RAG flow and measure it on the evaluation set. Iterate on retrieval, prompts and model choice against numbers, not impressions.
- Integrations and guardrails. Add read-only tools, handoff to the ticketing system, output checks and security testing, including prompt injection attempts.
- Pilot. Launch to a limited share of traffic or one channel, with daily review of conversations and a fast path for fixing knowledge gaps.
- Rollout. Expand traffic as resolution and repeat-contact metrics stabilise; add write actions only after the read-only version proves reliable.
- Operations. Weekly review of samples and escalations, monthly evaluation runs, a knowledge maintenance process and a budget line for the team that keeps it all current.
In our example, steps 1–6 fit into the 12 weeks of build; step 8 is the 0.25 FTE in the running costs. Budgeting only for the build is the most common reason chatbots degrade within a year. Our analysis of why agentic AI projects fail shows the same pattern at a larger scale.
How do you choose a chatbot development partner?
“Chatbot development services” is a crowded market: platform vendors, agencies, system integrators and freelancers all use the same words. The questions below separate partners who have run assistants in production from those who have built demos.
| Criterion | What a strong answer looks like | Red flag |
|---|---|---|
| Evaluation | Shows an evaluation set, metrics and how results changed between iterations in past projects | ”We test it manually before launch” |
| Knowledge maintenance | Proposes a process and owners for keeping content current, with tooling | Knowledge base treated as a one-off import |
| Guardrails | Describes deterministic checks for commitments, personal data and scope | ”The prompt tells the model not to do that” |
| Handoff | Designs escalation triggers and context transfer to your ticketing system | Handoff added “in a later phase” |
| Model strategy | Model-agnostic, with routing and a plan for switching | Locked to one model or to their own platform |
| Security | Tests prompt injection, least-privilege tool access, logging | No mention of prompt injection |
| Compliance | Covers AI Act Article 50 disclosure, GDPR retention and DPA with model providers | Compliance left entirely to the client |
| Cost transparency | Separates build, tokens, infrastructure and maintenance in the proposal | Single fixed price with no running-cost estimate |
| Metrics | Commits to resolution rate and repeat contacts, not only containment | Success defined as “number of conversations” |
| Ownership | You own prompts, evaluation sets, integrations and data | Everything lives inside a proprietary platform |
Score each criterion 0–2. A partner below 12 out of 20 is likely to deliver a good demo and a difficult first year.
The key takeaways:
- Choose the chatbot type by use case. Deterministic flows where mistakes are expensive, LLM + RAG for the long tail of questions, agentic actions only after read-only access proves reliable.
- Most of the work is around the model. Knowledge maintenance, guardrails, handoff and evaluation decide whether the bot is trusted.
- Tokens are not the main running cost. In our example they are about a quarter of the monthly bill; people keeping quality up are the largest item.
- Measure resolution, not containment. Klarna’s reversal shows what happens when volume and cost dominate the evaluation.
- The company answers for its bot. Air Canada’s case and the AI Act’s transparency duties since August 2026 make that explicit.
At ARDURA Consulting we have been building software for clients in Poland, Europe, the Middle East and the USA for more than ten years, and today many of those projects include LLM components — from internal knowledge assistants to customer service bots integrated with existing systems. Through our software development services we build chatbots that start from real conversation data, are evaluated against numbers and remain the client’s property, whichever model they run on.
If you are planning a chatbot and want to check the scope, architecture or budget before choosing a vendor, talk to our team. We can review your use case, estimate build and running costs and suggest where a chatbot will — and will not — pay off.