On-device AI privacy boundaries: what stays local, what leaves, what users control

A note summarizer runs its model on the iPhone. The app also uploads diagnostic attachments, syncs saved summaries, and offers a server fallback when local generation fails. Calling the whole feature "on-device AI" leaves out most of the privacy decisions.

The useful question is where the note goes during each part of that workflow. That includes the original text, retrieved context, generated output, and anything captured when the request fails.

For a production app, I would want that answer written down before choosing the fallback behavior. A faster response is a poor reason to send a private document somewhere the user never agreed to send it.

1. Map the full request path

Take a specific feature: summarize the note the user has open.

Its data flow might include:

• Reading the selected note from local storage.
• Retrieving related notes to give the model context.
• Passing that text to a local model.
• Saving the generated summary alongside the original.
• Syncing the saved record to another device.
• Recording a failure in a crash-reporting service.

Only one of those steps is inference. Local inference says nothing about whether the saved summary syncs or the error report contains the note.

Keep a short data-flow record next to the feature implementation. Name each recipient, the fields it receives, and the reason it needs them. Include your own backend, any proxy, the model provider, and diagnostic services. Check queues and temporary files too: a request waiting for retry is still a copy of the user's data.

A useful review can then ask a precise question: does this recipient need the full note body? If the answer is no, remove it from that payload.

2. Describe each execution route accurately

An app can support several model routes with different privacy properties.

A genuinely local route performs inference on the device. Model downloads and availability checks may still require a connection, so distinguish "this request is processed locally" from a promise that the app never contacts a server.

Private Cloud Compute is remote processing with Apple's documented protections. Apple's Apple Intelligence privacy notice says data processed there is used to fulfill the request, is not stored or made accessible to Apple, and is not retained by Private Cloud Compute. It also describes limited, content-free request metadata that Apple may collect. Those are specific claims about that system.

Your own backend or a third-party model API needs its own assessment. A proxy adds a recipient; it does not make the request local. Check what each service retains, whether content can be used for training, who can access it, and what deletion means under the configuration you actually use.

Keep "not used for training" and "not retained" separate in the UI and documentation. A service can have a no-training policy while retaining requests for another purpose. Do not copy one provider's privacy promise onto a different route.

3. Make local-only a policy the app enforces

If the user chooses local-only processing, a timeout must not turn that choice into a cloud request.

Represent processing preference separately from model availability. A device can be unable to run the local feature while the user's privacy preference remains unchanged. The resulting screen should offer a manual workflow, a smaller input, or an explicit choice to use remote processing.

For the note summarizer, a compact behavior contract could be:

Local-only selected:
- local generation is allowed
- remote generation is forbidden
- unavailable local model shows the manual editing flow
- local errors do not enqueue a remote retry
 
Remote processing allowed for this note:
- send only the reviewed note content
- exclude related notes and attachments unless separately selected
- keep the chosen recipient fixed for this request
- changing recipient requires a new disclosure and decision

Enforce this contract where outbound requests are created. A disabled cloud button does not protect a background retry worker or another screen that calls the same service.

The server needs an equivalent check for work it performs later. Scope authorization to the relevant user and operation, and reject a job whose permission has been withdrawn or whose approved scope no longer matches. Avoid a single account-wide aiEnabled flag that accidentally authorizes every future AI feature.

4. Ask before personal data crosses the boundary

Apple's App Review Guideline 5.1.2(i) requires clear disclosure of where personal data will be shared with third parties, including third-party AI, and explicit permission before sharing it.

For a document feature, put that decision where the user can understand the consequence. Explain which content will be sent and who receives it. If both your backend and an external model provider process the document, account for both in the disclosure.

An illustrative confirmation might say:

Send this note's text to our server and the named AI provider to generate a summary? Attachments and other notes will not be included.

That copy is only appropriate if the implementation does exactly that. Put the actual provider name in the product. Offer a clear way to keep working locally, and make retention details accessible without hiding the basic data-sharing decision in a privacy-policy link.

OS permission to read a photo or file does not, by itself, explain or authorize sending its contents to an AI service. A subscription purchase does not establish that permission either.

If you offer a persistent remote-processing preference, explain its scope and let the user revoke it. Ask again when the recipient or data scope changes materially. Treat this as a product control alongside the applicable privacy and legal requirements, not as a claim that one dialog resolves every obligation.

5. Minimize input before constructing the request

Start from the user's selection. Summarizing one note should not serialize the notebook because passing the database object was convenient.

Create a dedicated request representation containing only what the operation needs. Exclude account details, internal record identifiers, and unrelated metadata unless they serve a documented purpose. For an image task, consider whether a crop or locally extracted text is sufficient, then evaluate whether the smaller input still produces an acceptable result.

Retrieval deserves its own review. A local search can find sensitive material that the user never intended to include in the current request. Apply access and selection rules before returning those results to the model. If remote generation is available, the retrieval path must respect the approved remote scope too.

Automated redaction can reduce exposure, but it can miss identifiers in prose or leave enough context to identify someone. Describe the transformation accurately. Removing names from a document is insufficient evidence to label the document anonymous.

6. Keep tools and diagnostics inside the same policy

A local model can call a tool that sends data over the network. Review tool implementations as part of the processing route.

A tool that searches an external service should accept narrow arguments. Check its destination and permitted fields in application code. Do not let generated text choose an arbitrary upload URL or expand a read from one selected note to the entire library. Treat instructions embedded in imported documents as untrusted content.

Diagnostics need the same discipline. A failure report can expose a prompt even when inference stayed local. Review SDK breadcrumbs, attached files, and automatic request capture, as well as explicit logging calls.

For routine observability, prefer fixed categories such as execution route, operation type, duration, and failure code. Avoid note titles and raw model output in analytics fields. Generated summaries can contain the same personal information as their source documents.

If support needs a content-bearing diagnostic export, make it a separate, explicit flow with a review step. Decide its recipients and retention before adding an "include diagnostics" checkbox.

7. Define what cancel, revoke, and delete can do

Cancellation can stop your app from starting another request. Once data has been transmitted, it cannot guarantee that a remote service never received or processed it.

Revoking remote permission should prevent new requests and stop queued work that has not been sent. Recheck permission when a queued operation is dispatched; permission recorded when it was enqueued may be stale. Ignore late responses if the feature no longer has a valid place to apply them.

Deletion has a wider scope. Inventory saved summaries, cached inputs, server-side jobs, and provider-side artifacts where the service exposes them. Explain any retention limits or backup behavior in the privacy documentation. A local cache deletion cannot prove that a remote copy has been deleted.

Also review where results appear. A summary saved into a synced notebook, shown in a widget, or included in a notification has a different exposure surface from a temporary result on the open screen. Give those outputs the same care as the original note.

8. Test the boundary through observable behavior

Privacy tests should assert what the application sends and stores. A prompt asking the model to "keep data private" cannot enforce a network policy.

Use synthetic fixtures with recognizable marker strings. Record calls at the transport boundary and inspect the serialized payloads. The local-only tests should verify that the remote client receives no calls during normal generation, local failure, or automatic retry.

Cover the transitions where permission is easy to lose:

• The user declines remote processing after local generation fails.
• Remote permission is revoked while a job is queued.
• A configured provider changes after consent was recorded.
• Related notes are retrieved but excluded from the approved upload.
• An error occurs after sensitive text enters the workflow.
• A remote response arrives after cancellation or account sign-out.

Assert that marker strings are absent from diagnostics and unauthorized payloads. Separately inspect real device traffic and the backend's request handling with test data. A mock proving the app never called its remote client does not cover a diagnostic SDK with its own transport.

When releasing changes, compare the tested routes with the feature copy, privacy policy, and App Store privacy disclosures. A new fallback or SDK can change the data flow even when the visible screen barely changes.

I would ship a narrower local feature before shipping a silent remote fallback. If remote processing improves the result enough to justify the transfer, the product should be able to explain that choice plainly and work sensibly when the user declines.