Small Models, Real Business Value: Designing for On-Device AI

On-device artificial intelligence can make mobile software more private, responsive and useful without a reliable connection. Realising those benefits means choosing the right capabilities, understanding the hardware and designing around the user’s task.

I am currently designing a mobile application with on-device artificial intelligence.

The investigation started with some familiar technical questions. Which models could run on a phone? How much storage would they need? What capabilities could I deliver on both iOS and Android?

As the design developed, the more useful question became: what should happen locally to give the user an excellent experience?

Someone opening an app to complete a small task wants a useful result with minimal interruption. They should understand what the application is doing, retain control of their information and be able to recover when something goes wrong.

That is the standard I am designing towards. Model selection follows from it.

Start with the task

“Add AI” is too broad to be a useful requirement.

A mobile application might need to recognise speech, classify information, find relevant material, interpret an instruction or produce a summary. Those are different jobs, with different requirements.

An operation governed by a clear rule may need only conventional code. A narrow classification task may suit a small specialist model. Generating a useful explanation from several sources introduces a different level of complexity.

Treating every interaction as a conversation with one general-purpose model makes those distinctions harder to manage.

My approach is to separate the capabilities first. For each one, I want to understand the input, the expected result, the acceptable delay and the consequence of an error. That creates a much better basis for deciding whether the work belongs in ordinary application logic, a local model or an optional cloud service.

There is a direct connection with our deterministic reporting workflow: use AI where interpretation helps, and retain controlled processing where the operation is already well defined.

Useful AI does not always generate text

One of the most interesting areas is helping an application find relevant information.

An embedding model produces numerical representations of text that can support searches based on meaning. That is a different job from generating an answer. Google’s EmbeddingGemma, for example, is designed for retrieval and related tasks on devices including phones, tablets and laptops. It illustrates the scope for specialised local capabilities. Google’s EmbeddingGemma overview

Consider the difference between remembering an exact phrase and remembering roughly what something was about.

An exact name, reference number or technical term can make conventional search very effective. A less precise recollection may benefit from semantic search. A useful application needs to handle both situations.

The engineering question is how well it retrieves the material the user actually needs.

A fast result that misses the important document is still a poor result. Equally, a fluent summary can be misleading if it draws on the wrong sources.

That is why I am giving retrieval quality substantial attention. Before asking a model to explain something, the application needs a dependable way to find the evidence.

Responsiveness is more than model speed

Running inference locally removes the network round trip for that operation. It does not guarantee an instant response.

The application may need to load a model, prepare an input, retrieve information and assemble a result. The user experiences the whole sequence.

This changes what needs measuring.

A model benchmark can help compare components, but the product needs to be assessed from the moment the user makes a request to the moment something useful appears. Initial setup also matters. An application that performs well after preparation may still create a poor first impression if that preparation is slow or unexplained.

I am particularly interested in the balance between work done in advance and work done when requested. Preparing information can make later interactions faster, but it consumes storage, processing time and energy.

Saving a little storage is not automatically the best decision if it makes every interaction frustrating. Equally, improving one response is not a sufficient reason to make the rest of the phone sluggish.

Those trade-offs need testing with realistic material and repeated use.

Privacy needs an explicit boundary

Processing information on the device can reduce the need to transmit it elsewhere.

Google documents local processing and offline operation as benefits of its ML Kit generative AI APIs, alongside avoiding an additional server cost for each inference call. Those benefits apply to the supported local operations; they do not automatically describe every part of an application. Google’s ML Kit GenAI overview

An app may still synchronise files, create cloud backups, collect diagnostics or offer remote processing for selected features.

For the design I am considering, those are separate decisions.

Where is the information stored? Where does the AI process it? What is synchronised? What leaves the device if the user chooses a cloud-assisted feature?

The answers should be understandable without reading an architecture document.

If a local capability is unavailable, silently sending the request to a remote model would change the privacy boundary. I would rather make that choice explicit and provide a useful local alternative where practical.

Local processing also leaves important responsibilities with the application: protecting stored information, managing access and avoiding unnecessary sensitive content in logs.

Privacy is a property of the complete workflow.

Different phones create a product decision

“Runs on iOS” or “runs on Android” is not a complete statement about AI availability.

Apple’s Foundation Models documentation requires developers to check whether its on-device model is available and plan an alternative experience when it is not. A supported device may still be waiting for the model to become ready. Google likewise documents differences in device support and model versions across its local AI capabilities. Apple’s availability guidance, Google’s device-support guidance

That leaves a genuine product choice.

One option is to define a minimum hardware requirement and deliver a more consistent experience within it. Another is to support a wider range of devices with different capability levels. Both introduce compromises.

I am treating that choice as part of the user experience. People should know what their device supports before they depend on a feature.

Cross-platform consistency also needs careful definition. The same underlying model is one way to pursue it, but the outcome that matters is whether users receive comparable quality, clear behaviour and appropriate control. Different implementations may be acceptable if testing supports that conclusion.

The information should outlast the model

A model is an implementation choice that may change.

The user’s information needs a longer life.

One principle guiding my design is to distinguish the durable material a person creates from the indexes and other representations the application derives from it.

A search index may be rebuilt. A model may be replaced. Neither should require the user to recreate their original work.

However, rebuildable does not mean unimportant. Reconstructing an index can take time and affect battery use. Model changes can alter retrieval behaviour and need validation. User corrections and relationships also need preserving if they cannot be reproduced reliably.

This becomes especially relevant when information moves between devices. The design needs to establish what represents the user’s work, what represents local processing and how changes are reconciled.

It is the same ownership principle explored in Beyond the First Build, applied to the person using the product: useful intelligence should strengthen their control over information.

Cloud processing can have a defined role

An emphasis on local processing does not require every possible task to run on the phone.

A more demanding request may justify a cloud model, provided the benefit is clear and the user has agreed to that processing.

The useful question is how much information the task actually requires. Retrieving relevant material locally could allow a remote model to work with a limited selection rather than receiving an entire collection.

That still involves disclosure. A small selection can contain sensitive information, so reducing volume does not remove the need for appropriate controls.

Where an application generates an answer from retrieved material, I want the supporting sources to remain accessible. A user should be able to inspect what the response is based on.

References alone do not establish correctness. The answer must accurately reflect the material, preserve uncertainty and avoid turning an exploratory thought into a settled decision.

When the evidence is insufficient, showing the relevant material and explaining the limitation can be more useful than producing a confident answer.

Measure the experience on real workloads

I am still in the design stage. The benefits described here are objectives to validate.

My own accumulated material provides a useful starting point for evaluation because it contains the variation that tidy demonstration data often misses: long documents, repeated topics, changed terminology and decisions developed over time. It will need to be supplemented with other representative workloads.

The questions I want testing to answer include:

• How quickly does a useful result appear, including on the first request?
• Does the application retrieve the right material when the wording is imprecise?
• How much memory, storage and battery does repeated use require?
• What happens without a connection or while a model is unavailable?
• Can the user recognise and correct an error?
• Does performance remain acceptable as the information grows?

Cost belongs in that assessment too. Local inference can reduce recurring cloud expenditure, but development, device testing, model distribution and support still require investment. Processing also consumes resources on the user’s device.

The business case depends on whether those choices produce a better product at a sustainable overall cost.

Design intelligence around the person

Exploring on-device AI has brought me back to a familiar engineering principle: understand the task, then choose the tools that make it work well.

Small models create opportunities to place useful capabilities close to the user and their information. Realising that opportunity requires deliberate choices about responsiveness, evidence, device support, privacy and recovery.

My aim is an application that feels dependable in ordinary use: quick when the task is simple, clear when more work is required and honest when it cannot provide a reliable answer.

That is the experience the technology needs to earn.

If you are exploring where AI could improve a mobile product or business workflow, talk to virtco® about turning the idea into a practical, measurable design.