There are many layers of trust when it comes to AI. One of the most foundational is whether an AI is hallucinating or whether it’s answers can be trusted. But in this article, I want to focus on trust as it relates to confidentiality. Merriam Webster defines confidentiality as:
Intended for or restricted to the use of a particular person, group, or class.Merriam Webster
When we share the private details of our lives with a friend, we can trust that information will be kept secret. A future where AI could bare witness to every moment as it unfolds would require an unparalleled amount of trust in confidentiality.
Presently, I see two main challenges when it comes to confidentiality with AI:
In the past, LLMs couldn’t break confidentiality themselves because the only outputs they had were their conversations with the user. But the introduction of Tool Calling allows LLMs to reach outside their conversational sandboxes and interact with external systems. Even with the best intentions, this capability could open the door to accidental sharing of private information. (It’s worth noting that LLMs without Tool Calling don’t generally expose this risk, but the rise of “Agentic AI” will make this much more common.)
In my opinion, the bigger risk to confidentiality in AI right now are the rights the AI provider has to personal data. Licensing terms define these rights and configuration settings are intended to create limits, but often providers use language that’s hard for users to understand.
ChatGPT, for example, offers the setting “Improve the model for everyone”.
This setting (which is On by default) is worded in a way that encourages people to leave it on if they wish to continue being helpful to others. It takes digging down 3 more clicks to find the following statement:
We do not and will not use any personal information in training information to build profiles about people, to contact them, to advertise to them, to try to sell them anything, or to sell the information itself.
This is good news! But what will happen if this policy ever changes? If OpenAI were to start using chat data for profiling or advertising purposes, would “Improve the model for everyone” change to Off by default? And how would users get notified about this change?
Similar problems exist for developers who want to use AI services in their apps. Here’s a snippet from the Gemini API Pricing page:
“Used to improve our products” doesn’t sound bad, especially if developers can use the service for free right? But one click deeper we read:
Unpaid Services – To help with quality and improve our products, human reviewers may read, annotate, and process your API input and output. […] Do not submit sensitive, confidential, or personal information to the Unpaid Services.
In addition, Google offers a feature where users can get search results based on chat history. This feature, called “Grounding with Google Search”, has the same terms for both paid and free tiers:
[W]hen using Grounding with Google Search, Google will store prompts, contextual information that you may provide, and output for thirty (30) days for the purposes of creating Grounded Results and Search Suggestions and the stored information can be used for debugging and testing of systems that support Grounding with Google Search.
How Google uses data for debugging purposes is explained over 40 pages. It’s not easy reading.
Sort of. Most “Companion AI” scenarios require at least 4 models to function:
| Model | Usage |
|---|---|
| Speech Recognition | Understand what the user is saying and convert it into text for the LLM to process. |
| Image Understanding | Convert images (and eventually video) into context the LLM can understand. |
| Large Language (LLM) | Process written text and additional context to generate a response or do work. |
| Text to Speech | Speak responses and updates naturally back to the user. |
While any one of these models could theoretically run on a phone, each would be tightly constrained in capabilities.
Take Whisper, for example, a popular and capable speech recognition AI. Apps running Whisper locally on Android tend to use the Tiny version of the model. This makes sense from a RAM and speed perspective, but it also introduces serious limitations on accuracy.
| Size | Required RAM | Relative speed | English WER | Dutch WER |
|---|---|---|---|---|
| Tiny | ~273 MB | ~10x | 15.7 % | 39.4 % |
| Base | ~388 MB | ~7x | 11.7 % | 28.4 % |
| Small | ~852 MB | ~4x | 8.3 % | 17.2 % |
| Medium | ~2.1 GB | ~2x | 6.8 % | 11.7 % |
| Large | ~3.9 GB | 1x | 6.3 % | 10.2 % |
Word Error Rate (WER) is the percentage incorrectly recognized compared to what was spoken. Notably, Tiny is 9% less accurate in English and 28% less accurate in Dutch compared to the Medium model. At the same time, Medium requires 2 GB more RAM and is 5 times slower than Tiny. With speech recognition being only one of the four models necessary, running Companion AI scenarios locally isn’t feasible in the current hardware landscape.
With cloud hosting a necessity for Companion AI scenarios, the next logical question is “Which providers are likely to prioritize confidentiality?”
Historically, companies like Microsoft and Apple have been very privacy focused. Apple was first to push what’s possible with custom local AI silicon, and Microsoft eventually followed suit. These companies have always charge a premium for their software and services. Conversely, companies like Google and Facebook have offered services for free, then covered operating costs through advertising. Unfortunately, it’s no secret that advertising becomes more effective as more is known about the audience. Providers like ChatGPT and Claude make their money on the services they sell, though they’re still happy to train on your data if you let them.
History might show us how a provider tends to monetize, but free market customers can also speak with their wallets. Older generations (especially Gen X and beyond) place a high value on privacy and are usually willing to pay for it if there’s an option. Younger generations (especially Gen Z and below) are more willing to give up privacy if it can save on recurring fees. As AI continues to evolve, it won’t surprise me to see providers start offering different price points for different privacy needs.
For now, the best guidance I can offer is to read the fine print. And remember: If you’re not paying for the product, chances are you are the product.
I believe the single greatest step for providers to earn trust would be providing transparent details about how personal information is used. I’d love to see every provider offer a table like the following for their app or service:
| Can information I share ever be connected to my real-world identity? | ✅ No | ⚠️ Yes |
|---|---|---|
| Are personally identifiable details from my interactions (such as names, locations and addresses) ever stored? | ✅ No | ⚠️ Yes |
| Can my messages, data or account information ever be used for marketing or advertising? | ✅ No | ⚠️ Yes |
| Will my messages, data or account information ever be used to train or improve models? | ✅ No | ⚠️ Yes |
| Will my messages, data, or account information ever be shared with a 3rd party? | ✅ No | ⚠️ Yes |
| Can a human whom I’ve not given permission ever read my messages or view my data? | ✅ No | ⚠️ Yes 🅰️ |
| How long are my messages and data stored? | ??? | ??? |
Example: 🅰️ A human may review your messages or data in the following circumstances: Detection of an unlawful act, complaint filed by another user, government petition to support legal proceedings, …
I believe the second greatest step providers could take to earn trust would be leveraging independent 3rd parties to verify claims like the ones above.
I see us on the edge of a computing evolution. One that will fundamentally change how humans interact with computers and each other. Companion AI has the potential to radically improve quality of life while simultaneously reducing distractions and allowing us to be more present. But these advances are only possible if we trust AI. Trust it to be present with us, to watch lives unfold, and to be our true companions.
