Cost, latency and model selection trade-offs

Lorraine Beaudry had waited nine lessons for her number. "What will the overnight summaries cost, and could a cheaper model do the job?" she asked as the tenth AI hour began. Dr. Kaur had a different number in mind: how long the summaries would take to appear. Gwen wanted to know whether the service would be available if one data centre had a bad night.

Noor said all three questions belonged together, because cost, speed, quality and availability trade against each other in every generative AI decision. This lesson explains how token-based pricing works and what drives the bill, how latency is measured, the consumption options AWS offers for Amazon Bedrock and their trade-offs, and a structured way to choose a model using the selection factors in the exam guide: model type, performance requirements, capabilities, constraints, compliance, cost, latency and complexity.

How token-based pricing works

Most foundation model services, including on-demand use of Amazon Bedrock, charge per token, as lesson 2 introduced. Four facts about the pricing model explain most bills.

• Input and output are priced separately. Every token you send, including instructions, examples, retrieved documents and conversation history, is an input token. Every token the model writes is an output token. Output tokens usually cost several times as much as input tokens.
• Prices differ widely between models. A small, fast model can cost a small fraction of what the largest model from the same provider costs per token.
• Prices are quoted per thousand or per million tokens, so the arithmetic needs care with units.
• Other features add charges. Knowledge Bases storage and retrieval, Guardrails checks, embeddings, model customization and hosting of custom models all have their own prices.

Because input includes everything in the context, the biggest savings often come from context engineering rather than from bargaining over price: sending three relevant chunks instead of thirty pages, trimming instructions, and asking for concise output.

# Monthly cost estimate for overnight call summaries. All prices are ILLUSTRATIVE.
summaries_per_month = 3000
in_tokens = 250 + 150 + 500      # instructions + two examples + the call note
out_tokens = 180

models = {   # (input, output) USD per 1,000 tokens, illustrative only
    "small model": (0.00006, 0.00024),
    "mid-size model": (0.0008, 0.0032),
    "large model": (0.003, 0.015),
}
for name, (p_in, p_out) in models.items():
    per_call = in_tokens / 1000 * p_in + out_tokens / 1000 * p_out
    print(f"{name:15} per summary {per_call:.5f} USD, per month {per_call * summaries_per_month:.2f} USD")

small model     per summary 0.00010 USD, per month 0.29 USD
mid-size model  per summary 0.00130 USD, per month 3.89 USD
large model     per summary 0.00540 USD, per month 16.20 USD

Lorraine was surprised by how small the numbers were at the group's volume. Noor pointed out the two ways such costs grow unnoticed: volume, if every clinic starts using a tool for everything, and context, if someone adds an entire medical history to every request. Lorraine asked for a monthly budget alert on the account, which Gwen set up with AWS Budgets and cost allocation tags for the AI project.

Latency

Latency is how long a user waits. For generative models it has two parts that are worth separating.

• Time to first token: how long until the response starts appearing. It depends on the model, the load, and how much input must be processed first, so long contexts increase it.
• Output speed: how many tokens per second the model produces once it starts. Since output is generated one token at a time, a long answer takes proportionally longer.

Streaming sends tokens to the user as they are generated, so people see the answer start quickly even if it takes several seconds to finish. Chat interfaces almost always stream. The overnight summaries do not need streaming at all, because nobody is waiting for them in real time; they need to be ready by the morning handover.

# Rough response time = time to first token + output tokens / output speed. Illustrative values.
cases = [("small model", 0.3, 150), ("mid-size model", 0.6, 90), ("large model", 1.2, 50)]
for out_tokens in (60, 400):
    for name, ttft, tok_per_s in cases:
        total = ttft + out_tokens / tok_per_s
        print(f"{out_tokens:3} tokens, {name:15}: about {total:.1f} s")

 60 tokens, small model    : about 0.7 s
 60 tokens, mid-size model : about 1.3 s
 60 tokens, large model    : about 2.4 s
400 tokens, small model    : about 3.0 s
400 tokens, mid-size model : about 5.0 s
400 tokens, large model    : about 9.2 s

Consumption options in Amazon Bedrock

Bedrock offers several ways to pay for and run inference. Each trades cost against responsiveness, guaranteed capacity or flexibility.

For the overnight summaries, batch inference was the natural fit: the notes accumulate overnight, and a batch job at the end of the shift can produce all the summaries before handover at a lower price. For the clinic managers' manual questions, on-demand with streaming suits people who are waiting at a screen. Provisioned Throughput would make sense only at much higher volume, or if the group fine-tuned a model, because customized models typically need dedicated capacity to run.

Gwen's availability question had an answer in the same table. Cross-Region inference spreads requests over several Regions, which improves resilience and throughput, but it sends processing to the Regions in the chosen profile. For Cascade Ridge on the Canadian site, that is acceptable only for a profile that stays within Canada; otherwise the group accepts in-Region capacity and plans a fallback of reading the full notes.

Other cost levers

• Choose the smallest model that meets the bar. Test a small model first and move up only if it fails the evaluation.
• Reuse repeated context. Bedrock supports prompt caching for supported models, which reduces the cost and latency of sending the same long instructions or documents repeatedly.
• Route by difficulty. Intelligent prompt routing in Bedrock can send each request to a cheaper or more capable model in the same family depending on the request; a team can also route simple tasks to a small model in its own code.
• Limit output. A maximum output length and a request for concise answers cut the most expensive tokens.
• Customize carefully. Fine-tuning and distillation can let a smaller model do a specialized job well, but training and hosting add their own costs.

Choosing a model

The exam guide lists the factors to weigh when selecting a generative AI model. Noor turned them into questions for the summaries.

The illustrative chart makes a point the team found in its own tests: the gap between models often shrinks when the context is good. A small model with clear instructions and the right retrieved passages may meet the bar that a large model meets without them, at a fraction of the cost.

# Pick the cheapest model that passes every hard requirement. Illustrative test results.
candidates = [
    # name, eval score (of 50), monthly cost USD, in our Region, ready by handover
    ("small model", 41, 1.2, True, True),
    ("mid-size model", 47, 12.0, True, True),
    ("large model", 49, 55.0, False, True),
]
required_score = 46
ok = [c for c in candidates if c[1] >= required_score and c[3] and c[4]]
best = min(ok, key=lambda c: c[2])
print("passing:", [c[0] for c in ok])
print("choose:", best[0], "at about", best[2], "USD a month")

passing: ['mid-size model']
choose: mid-size model at about 12.0 USD a month

In the illustrative results, the large model scored highest but was not offered in the group's Region, so it failed a hard constraint before cost was even considered. The mid-size model passed every requirement at modest cost. Noor reminded Jonah that the choice is provisional: the evaluation is re-run when new models or versions appear, and the group can switch with little work because Bedrock uses one API.

Watching costs after launch

Estimates are only the start. Once a tool is live, Gwen's standard AWS cost practices apply to generative AI as to anything else. Cost allocation tags on every Bedrock resource and application let Lorraine see the AI project's spending separately in Cost Explorer. AWS Budgets sends an alert when monthly spending passes a threshold, and Cost Anomaly Detection flags unusual jumps, such as a looping script calling a model thousands of times overnight. Bedrock invocation metrics in Amazon CloudWatch show the number of requests and tokens, so a cost change can be traced to its cause: more users, longer contexts, or longer answers.

Jonah added a simple efficiency metric to his weekly report: average input tokens and output tokens per request. If the average input grows, someone has probably added material to the context; if output grows, the length limit or the instructions may have drifted. Both are cheap to fix once noticed. Noor called this the generative AI version of checking the electricity meter: nobody needs to watch it every day, but everyone should know where it is and what normal looks like.

Balancing all four

Cost, latency, quality and availability rarely all improve together. A larger model raises quality and cost and usually latency. Batch processing lowers cost and removes any real-time response. Cross-Region inference raises availability but may conflict with data residency. Streaming improves perceived latency without changing cost. Good decisions start from the task's requirements, treat some of them, such as quality thresholds and data residency, as fixed, and then optimize the rest. For the overnight summaries, the fixed requirements were the clinical quality bar, the Region and readiness by handover; cost was then minimized within those limits.