Beyond the desktop: mobile, voice, XR and AI interfaces

Most HCI theory was developed for people sitting at desks with keyboards, mice and large screens. Today most interaction happens elsewhere: on phones held in one hand on a moving bus, by voice in kitchens and cars, through watches and earbuds, in head-mounted displays, and increasingly through conversation with AI systems that act on the user's behalf. The principles of earlier lessons still apply, but each modality brings its own constraints and design knowledge. This lesson surveys mobile and touch interaction, voice and conversational interfaces, extended reality, wearables, in-vehicle interfaces and, at more length, the design of interactions with AI systems, including large language model (LLM) interfaces, as of 2026. AI-410 covers how large language models work, AI-430 their evaluation and safety and ROB-430 human-robot interaction; this lesson focuses on the human side of the interface.

Mobile and touch

Phones impose small screens, imprecise touch input, frequent interruption and highly variable contexts: bright sun, rain, walking, one-handed use, poor connectivity. Design responses covered earlier include large touch targets and thumb-reachable primary actions (lesson 3), glanceable layouts with strong visual hierarchy (lesson 7) and reflow at large text sizes (lesson 10). Gestures add expressiveness but have a discoverability problem: a swipe-to-delete or long-press menu has no visible signifier, so gestures should be shortcuts for actions also available through visible controls, and they need single-pointer alternatives (WCAG 2.5.1 and 2.5.7). Mobile users also face slow and costly networks: RideLine's redesign caches the user's saved stops and passes so that proof of payment works offline, and loads the arrivals list before the map. Notifications deserve restraint, because each interruption has a cost, and users who are notified too often turn notifications off altogether; service disruption alerts for a rider's saved routes earned their place, promotional messages did not.

Voice and conversational interfaces

Voice user interfaces (VUIs), such as smart speakers, phone menus and in-car assistants, offer hands-free and eyes-free interaction, help people who cannot easily use screens and can be fast for simple commands. They also have fundamental limits. Speech is transient and linear: users cannot skim a spoken list or go back to an earlier option, so lists must be short, with the most likely items first, and working memory (lesson 2) is the bottleneck. Voice interfaces have almost no visible signifiers, so users do not know what they can say, a severe gulf of execution. Recognition errors are inevitable, so dialogue design must confirm important actions and make errors easy to repair, using implicit confirmation for low-risk actions ('Next departure from Main and 5th is at 7:42') and explicit confirmation for consequential ones ('Buy a monthly pass for 98 dollars?').

Good conversational design borrows from the study of human conversation. Paul Grice's cooperative principle and maxims of quantity (say as much as needed and no more), quality (say what is true), relation (be relevant) and manner (be clear and brief) are a useful checklist for system prompts and responses. Turn-taking cues tell users when the system is listening. Persona and tone should match the context and stay consistent. Users should always be able to ask for help, repeat and exit. Error messages should progressively add help: after a first failure, reprompt briefly; after a second, give examples of what can be said; after a third, offer another channel. Voice interfaces also raise privacy concerns, since always-listening devices in homes and workplaces capture speech that users may not intend to share.

Extended reality

Extended reality (XR) covers virtual reality (VR), which replaces the user's view with a computer-generated environment, augmented reality (AR), which overlays digital content on the physical world through a phone or see-through headset, and mixed reality, in which virtual objects interact with the physical environment. XR interaction relies on head and eye tracking, hand tracking, controllers and voice. It also brings human factors problems of its own. Cybersickness, a form of motion sickness caused by conflict between visual motion and the vestibular sense, affects a substantial share of users, especially with artificial locomotion; designers use teleportation, stable reference frames, vignetting during movement and high, steady frame rates to reduce it. The vergence–accommodation conflict, in which the eyes converge on a virtual object at one distance while focusing on a display at another, causes fatigue. Text is harder to read, arms tire when held up to interact (the 'gorilla arm' effect known from vertical touch screens), and users may collide with real objects. XR accessibility is an active area: interfaces that assume two hands, binocular vision, standing posture or hearing exclude many users, and W3C's XR Accessibility User Requirements note collects needs and approaches.

AR is closer to everyday use than VR for applications like RideLine. A prototype that overlaid walking directions and the platform for a transfer on the phone's camera view tested well with visitors such as Daniel, but walking while looking through a phone raised obvious safety concerns, and the team limited the feature to station interiors with a reminder to look up.

Wearables and ambient interfaces

Smartwatches, fitness bands, earbuds and smart glasses offer micro-interactions measured in seconds. Their screens are tiny or absent, so they favor glanceable information, haptic and audio feedback, and actions completed in one or two taps. A watch complication showing the next departure from the rider's usual stop, with a haptic tap when it is time to leave, suits the medium; buying a fare on a watch is better done by showing a pass already bought on the phone. Mark Weiser's vision of ubiquitous computing (1991) and calm technology (with John Seely Brown, 1996) argued that the most profound technologies disappear into the background, moving between the periphery and the center of attention only when needed, a useful ideal for notifications and ambient displays.

In-vehicle interfaces

Interfaces in cars are a special case because the user's primary task is driving, and distraction can kill. Driver distraction research distinguishes visual (eyes off the road), manual (hands off the wheel) and cognitive (mind off driving) distraction. Studies of naturalistic driving, in which instrumented cars record ordinary trips, found that eyes-off-road glances longer than about 2 seconds sharply increase crash and near-crash risk. In 2013 the US National Highway Traffic Safety Administration (NHTSA) published voluntary visual-manual driver distraction guidelines for in-vehicle electronic devices, which recommend that tasks performed while driving be designed so that, in a test with 24 drivers in a driving simulator, at least 21 drivers keep 85 percent of their glances at the device to 2 seconds or less, a mean glance duration of 2 seconds or less, and a total eyes-off-road time of 12 seconds or less per task; tasks that fail should be locked out while the vehicle moves. Transport Canada has published guidance on driver interaction with in-vehicle displays based on similar principles. Voice interaction reduces visual-manual demand but not cognitive load, which also impairs driving, a point several studies of voice assistants in cars have made.

# Checking an in-vehicle task against the NHTSA visual-manual distraction guideline criteria
# (2013): for each of 24 test drivers, 85% of glances at the display <= 2.0 s, mean glance <= 2.0 s,
# total eyes-off-road time <= 12 s. Invented, seeded glance data for "find the next departure"
# on a car-dashboard version of RideLine, before and after simplifying the screen.
import random, statistics as st
random.seed(421)
def drivers(n_glances, mean_len):
    return [[max(0.3, random.lognormvariate(0, 0.35) * mean_len) for _ in range(random.randint(*n_glances))]
            for _ in range(24)]
for name, data in (("original screen", drivers((6, 10), 1.5)), ("simplified screen", drivers((3, 4), 0.9))):
    pass_count = 0
    for g in data:
        ok = (sum(x <= 2.0 for x in g) / len(g) >= 0.85, st.mean(g) <= 2.0, sum(g) <= 12.0)
        pass_count += all(ok)
    all_glances = [x for g in data for x in g]
    print(f"{name:18} mean glance {st.mean(all_glances):.2f} s, mean total {st.mean(sum(g) for g in data):.1f} s, "
          f"glances over 2 s {sum(x > 2 for x in all_glances) / len(all_glances):.0%}, drivers meeting all criteria {pass_count}/24")
print("(the guideline asks that at least 21 of 24 drivers meet the criteria)")

original screen    mean glance 1.55 s, mean total 12.1 s, glances over 2 s 19%, drivers meeting all criteria 9/24
simplified screen  mean glance 0.99 s, mean total 3.4 s, glances over 2 s 0%, drivers meeting all criteria 24/24
(the guideline asks that at least 21 of 24 drivers meet the criteria)

The original dashboard screen, which required scrolling a list of departures, fails badly: only 9 of 24 drivers meet all three criteria, because nearly a fifth of glances run past 2 seconds while drivers read and scroll, and the average total eyes-off-road time sits right at the 12-second limit even though the mean glance is well under 2 seconds. The simplified screen, which shows only the next two departures from the stop nearest the destination, needs three or four short glances, and every simulated driver meets the criteria. The test illustrates a general point: the total demand of a task matters as much as the length of each glance, and the best in-vehicle design is often to remove the task from the driving context altogether.

Designing interaction with AI systems

AI-based features differ from conventional ones in ways that matter for interaction. Their outputs are probabilistic and sometimes wrong in ways that are hard to predict; their behavior may change as models are updated; their reasoning is often opaque; and they increasingly act, not just advise. Saleema Amershi and colleagues at Microsoft synthesized two decades of guidance into 18 guidelines for human-AI interaction (CHI 2019), grouped by phase. Initially: make clear what the system can do and how well it can do it. During interaction: time services based on context, show contextually relevant information, match social norms and mitigate social biases. When wrong: support efficient invocation, dismissal and correction, scope services when in doubt, and make clear why the system did what it did. Over time: remember recent interactions, learn from user behavior cautiously, update and adapt cautiously, encourage granular feedback, convey the consequences of user actions, provide global controls and notify users about changes.

Trust is central. John Lee and Katrina See's influential review (2004) argued that the goal is not maximal trust but calibrated trust, in which reliance matches the system's actual reliability. Over-trust leads to automation bias, the tendency to accept automated recommendations without sufficient scrutiny, including errors of commission (following a wrong suggestion) and omission (missing a problem the system failed to flag); Raja Parasuraman and Dietrich Manzey's 2010 review documented automation bias among pilots, clinicians and process operators. Under-trust leads to disuse of systems that would help. Interfaces support calibration by communicating uncertainty honestly, explaining the basis for outputs where possible, showing track records, and making it easy to verify and override.

Explanation is a large research area, explainable AI (XAI). Explanations can be global (how the model works in general) or local (why this output), and can take the form of feature attributions, examples, counterfactuals ('your trip would be shown as on time if the connecting train were not delayed') or natural language rationales. Studies of explanations in decision support have produced a sobering finding: explanations can increase users' acceptance of AI recommendations whether or not the recommendations are correct, which can worsen over-reliance, as work by Gagan Bansal, Zana Buçinca and colleagues around 2021 showed. Interventions that ask users to commit to their own judgment first, or that slow down acceptance (cognitive forcing functions), can reduce over-reliance at some cost in convenience. Explanations should be designed and evaluated for whether they help users decide correctly, not for whether users like them.

Mixed initiative

When should a system act on its own, ask, or stay quiet? Eric Horvitz's 'Principles of mixed-initiative user interfaces' (1999) framed this as a decision under uncertainty: the system estimates the probability that the user wants an action and compares the expected utility of acting automatically, engaging the user in a dialogue, or doing nothing, given the costs of each outcome. This yields thresholds: below a low probability, do nothing; above a high one, act; in between, ask. The thresholds depend on the costs, especially the cost of an unwanted action and how easily it can be undone. The program below applies the idea to an invented RideLine assistant deciding whether to renew a rider's monthly pass.

# Mixed-initiative decisions (after Horvitz, 1999): should an assistant act automatically,
# ask the user, or do nothing, given its confidence p that the user wants the action?
# Utilities are invented for RideLine "renew my monthly pass now?" on the last day of the month.
U = {  # (decision, user actually wants it?) -> utility to the user
    ("act", True): 1.0, ("act", False): -2.0,     # an unwanted purchase is costly to undo
    ("ask", True): 0.7, ("ask", False): -0.2,     # asking costs a little attention either way
    ("none", True): -0.5, ("none", False): 0.0,   # missing a wanted renewal is annoying
}
def best(p):
    eu = {d: p * U[(d, True)] + (1 - p) * U[(d, False)] for d in ("act", "ask", "none")}
    return max(eu, key=eu.get), eu
for p in (0.05, 0.2, 0.4, 0.6, 0.8, 0.9, 0.97):
    d, eu = best(p)
    print(f"p = {p:.2f}: act {eu['act']:+.2f}, ask {eu['ask']:+.2f}, none {eu['none']:+.2f} -> {d}")
# thresholds where the best choice changes, solved from equal expected utilities
p_none_ask = (U[("none", False)] - U[("ask", False)]) / ((U[("ask", True)] - U[("ask", False)]) - (U[("none", True)] - U[("none", False)]))
p_ask_act = (U[("ask", False)] - U[("act", False)]) / ((U[("act", True)] - U[("act", False)]) - (U[("ask", True)] - U[("ask", False)]))
print(f"do nothing below p = {p_none_ask:.2f}, ask between, act automatically above p = {p_ask_act:.2f}")

p = 0.05: act -1.85, ask -0.15, none -0.03 -> none
p = 0.20: act -1.40, ask -0.02, none -0.10 -> ask
p = 0.40: act -0.80, ask +0.16, none -0.20 -> ask
p = 0.60: act -0.20, ask +0.34, none -0.30 -> ask
p = 0.80: act +0.40, ask +0.52, none -0.40 -> ask
p = 0.90: act +0.70, ask +0.61, none -0.45 -> act
p = 0.97: act +0.91, ask +0.67, none -0.48 -> act
do nothing below p = 0.14, ask between, act automatically above p = 0.86

Because an unwanted purchase is costly, the assistant should act automatically only when it is very confident and otherwise ask; when it has little reason to think the rider wants to renew, it should stay quiet rather than nag. Making the purchase easy to undo would lower the cost of a wrong action and move the act threshold down, which is one reason undo is so valuable in AI features. The framework also makes explicit something designers often leave implicit: the assistant's confidence must itself be well calibrated, or the thresholds are meaningless.

Interfaces to large language models

Since the public release of ChatGPT in November 2022, conversational interfaces to large language models have become one of the most widely used interaction styles in computing, and by 2026 they are embedded in search, productivity software, programming tools and customer service, and increasingly operate as agents that take multi-step actions such as booking, purchasing or editing files. They inherit both the strengths of natural language (no syntax to learn, flexible requests) and the weaknesses of conversational interfaces (poor discoverability of capabilities, transient output), and they add problems of their own. LLMs can produce fluent, confident text that is false, often called hallucination or confabulation, and their fluency invites over-trust. Their capabilities are uneven and not obvious, so users form inaccurate mental models of what they can do. And prompt-based interaction shifts effort to articulating intent, which some users find hard.

Design practice has converged on several responses, though evidence is still accumulating as of 2026. Ground answers in sources and show citations that users can check, as RideLine's conversational trip planner does by linking every claim about service to the agency's live data and showing the time it was retrieved. Express uncertainty in language and visual cues, and avoid false precision. Offer suggested prompts and examples to bridge the gulf of execution. Keep humans in control of consequential actions with previews and confirmation, and log what agents did so it can be reviewed and undone. Provide structured interfaces (buttons, forms, tables) alongside chat where tasks are well defined, since chat is a poor interface for selecting among many options. Disclose clearly when users are interacting with an AI rather than a person, a requirement under some laws such as the European Union's AI Act transparency obligations and several US state laws, and an ethical baseline everywhere. CMPUT-466 and AI-410 discuss the underlying models, and AI-440 the evolving law.

Worked examples