Application integration: SQS, SNS and EventBridge

Kwame has a confession. BookVet's reminder texts are sent by a loop inside the web application. Every evening, when a scheduled task runs, the loop reads tomorrow's appointments and sends a text for each one through a messaging provider. Twice last year the provider was slow, the loop timed out halfway, and some clients got two reminders while others got none. Once, the whole web application slowed down for everyone because it was busy sending texts. "Everything is tied to everything," he says.

Gwen calls this tight coupling: when parts of a system call each other directly and wait for each other, a slow or failed part drags the others down. The cure is loose coupling, in which parts communicate through messages and events, so that each can fail, slow down or scale without breaking the rest. AWS offers several managed services for this. The exam focuses on three: Amazon SQS for queues, Amazon SNS for notifications and Amazon EventBridge for events. This lesson also covers the business application services the exam lists in the same task, Amazon SES and Amazon Connect.

Amazon SQS: queues

Amazon Simple Queue Service (Amazon SQS) is a fully managed message queue. A producer sends messages to a queue; a consumer polls the queue, receives messages, processes them and deletes them. The queue holds messages until they are processed, so producer and consumer do not have to run at the same time or at the same speed. If a thousand bookings arrive in a minute and the workers can handle a hundred a minute, the queue simply grows and then drains; nothing is lost and the web front end stays fast.

Several SQS ideas come up repeatedly.

• Standard queues offer nearly unlimited throughput with at-least-once delivery and best-effort ordering, so a consumer must cope with an occasional duplicate or out-of-order message.
• FIFO queues (first in, first out) guarantee order and exactly-once processing within a message group, at lower throughput. Use them when order matters, such as steps in a payment.
• The visibility timeout hides a message from other consumers while one consumer works on it. If the consumer finishes, it deletes the message. If it crashes, the timeout expires and the message becomes visible again so another consumer can try.
• A dead-letter queue receives messages that have failed processing a set number of times, so a single bad message does not block the queue forever and can be inspected later.
• Messages are kept for up to 14 days (four days by default), and each message can be up to 256 KB, with larger payloads stored in S3 and referenced from the message.

In SQS, each message is normally processed by one consumer. Queues spread work across a pool of workers; they are not for telling many systems about the same thing. That job belongs to SNS.

# A tiny queue simulation: visibility timeout and a dead-letter queue after 3 failed receives.
from collections import deque

queue = deque([{"id": 1, "body": "remind Biscuit", "receives": 0},
               {"id": 2, "body": "remind Pepper (bad phone number)", "receives": 0},
               {"id": 3, "body": "remind Luna", "receives": 0}])
dead_letter, sent = [], []
MAX_RECEIVES = 3

def process(msg):
    return "bad phone" not in msg["body"]      # pretend the SMS provider rejects bad numbers

while queue:
    msg = queue.popleft()
    msg["receives"] += 1
    if process(msg):
        sent.append(msg["id"])                    # success: consumer deletes the message
    elif msg["receives"] >= MAX_RECEIVES:
        dead_letter.append(msg["id"])             # moved to the dead-letter queue
    else:
        queue.append(msg)                         # visibility timeout expires; retried later

print("sent:", sent)
print("dead-letter queue:", dead_letter)

sent: [1, 3]
dead-letter queue: [2]

Kwame's redesign puts each reminder in a queue as its own message. A Lambda function reads from the queue and sends one text per message. If the messaging provider is slow, messages wait; if a send fails, the message is retried; if a phone number is simply wrong, the message lands in the dead-letter queue after three attempts and the clinic gets a daily list of numbers to correct. The web application is no longer involved at all.

Amazon SNS: publish and subscribe

Amazon Simple Notification Service (Amazon SNS) is a managed publish-subscribe (pub/sub) service. A publisher sends a message to a topic; every subscriber to the topic receives a copy. Subscribers can be email addresses, mobile phone numbers for SMS, mobile app push notifications, HTTPS endpoints, Lambda functions and SQS queues. Sending one message to many receivers at once is called fan-out.

The exam often pairs SNS with alerts and notifications. CloudWatch alarms, from lesson 7, publish to SNS topics, and the topic delivers the alert to the on-call phone, an email list and a chat integration all at once. A common pattern combines the two services: an SNS topic fans out to several SQS queues, each feeding a different system, so that each system gets its own copy of every message and works through it at its own pace.

For Cascade Ridge, Gwen creates an SNS topic named practice-system-alerts. The VetLedger canary alarm publishes to it, and its subscribers are Mateo's phone by SMS, the IT email address, and, at Dr. Kaur's request, the emergency hospital's charge nurse phone, so the night team knows to use paper fallback forms.

Amazon EventBridge: routing events

An event is a record that something happened: an order was placed, an instance changed state, a file was uploaded, a payment failed. Amazon EventBridge is a serverless event bus that receives events and uses rules to match them by their content and send them to targets, such as Lambda functions, Step Functions workflows, SQS queues, SNS topics or API destinations in other software.

EventBridge receives events from three kinds of source: AWS services (many send events automatically, such as an EC2 instance stopping or a CodePipeline release failing), your own applications, and software-as-a-service partners that integrate with it. Because rules filter on content, one bus can route many kinds of event to many targets: payment failures to the billing team's queue, high-value orders to a fraud check and every event to an archive.

Two further features matter for Cascade Ridge. EventBridge Scheduler runs targets on a schedule, from once at a set time to every few minutes, replacing old scheduled tasks on servers. EventBridge Pipes connects a source, such as a queue or a DynamoDB stream, to a target with optional filtering and enrichment, without writing code to glue them together.

The evening reminder job becomes a schedule in EventBridge Scheduler that starts a Lambda function each evening, which reads tomorrow's appointments and puts one message per reminder into the SQS queue. When VetLedger records a new vaccination, an event on a custom bus can trigger the function that schedules the next vaccination reminder months ahead.

Other integration services to recognize

• AWS Step Functions (lesson 2) coordinates multi-step workflows with retries and branching. Prescription refills, which need a vet's approval before the pharmacy fills them, become a Step Functions workflow that waits for the approval.
• Amazon MQ is a managed message broker for Apache ActiveMQ and RabbitMQ, chosen when an existing application already uses those standard protocols and cannot easily be changed to SQS and SNS.
• Amazon API Gateway (lesson 2) is the front door for request-response APIs, which complements message-based integration.

Business application services: Amazon SES and Amazon Connect

The exam lists two business application services alongside integration. Amazon Simple Email Service (Amazon SES) sends large volumes of email from applications, such as booking confirmations, invoices and newsletters, and can receive email too. It handles the reputation and deliverability work, such as authenticating the sending domain, that makes email arrive in inboxes rather than spam folders. SNS can send simple email notifications to subscribers, but customer-facing, branded email at volume is SES's job.

Amazon Connect is a cloud contact center service: phone and chat queues, interactive voice menus, call routing and recording, paid per use with no telephone hardware. Cascade Ridge's phones are already a hosted voice service, which the migration plan retains, but Andre is interested, because clinics still miss calls at opening time and Connect could route overflow calls from a busy clinic to a central booking team. Gwen records it as an idea for later, not part of this project.

Designing for duplicates and retries

Loose coupling brings a new responsibility. Because standard SQS queues deliver each message at least once, and because retries happen whenever a consumer fails part way through, a consumer may occasionally see the same message twice. Sending a client two identical reminder texts is a small annoyance. Charging a client twice, or recording the same vaccination twice, is not. The cure is to make processing idempotent: doing it twice has the same effect as doing it once.

Kwame's reminder function does this with a simple check. Each message carries the appointment number and the reminder type. Before sending a text, the function records that pair in a DynamoDB table using a write that succeeds only if the pair is not already there. If the write fails because the pair exists, the reminder has already been sent and the function simply deletes the message. The same idea, a unique key checked before acting, protects the refill workflow and any future payment step.

Gwen adds two settings to the design. The queue's visibility timeout is set longer than the function's maximum run time, so that a message is not handed to a second copy of the function while the first is still working on it. And an alarm watches the number of messages in the dead-letter queue, so that a sudden rise, perhaps because the messaging provider changed its interface, is noticed the same evening rather than when clients start calling to ask why their reminders stopped.

Andre asks how the clinics will know what happened to their reminders. CloudWatch metrics from the queue and the function show how many messages were sent, retried and moved to the dead-letter queue each night, and a short daily summary email goes to each clinic manager. For the first time, the clinics can see the reminder system working rather than assuming it is.

Gwen sums up the lesson with one sentence for Mateo's notes: queues hold work, topics broadcast news, and buses route events by what they say. Most integration questions on the exam can be answered by deciding which of those three a scenario describes.