AI in healthcare: how agents handle sensitive patient data

What the law says about health data

Privacy laws put health data in a protected category, alongside genetic and biometric data. For this kind of data, the conditions that allow processing are narrower than for ordinary personal data.

In healthcare, the most common grounds are processing for the provision of care by health professionals and services, compliance with a legal or regulatory obligation, and the patient’s explicit consent for specific purposes. In the US, HIPAA allows covered entities to use PHI for treatment, payment and healthcare operations, and requires a business associate agreement with any vendor that handles it. Which ground applies to each routine is a decision for the organization’s data protection officer and legal team, not for the technology vendor.

Some points deserve special attention in AI projects:

• Sharing health data for commercial gain is tightly restricted
• Health insurers face limits on using health data to select risks when accepting or excluding members
• The organization may need a data protection impact assessment for higher-risk processing
• Patients have the right to know what data is processed and why

On top of privacy laws, the sector has its own rules. Professional confidentiality is in the ethics codes of the health professions, and medical records are governed by specific regulations in each country. Data protection authorities and health regulators supervise and issue guidance on the subject.

Where the agent helps without touching clinical work

A large share of the repetitive work at a healthcare organization is administrative. An agent can take on these routines while touching as little clinical information as possible:

• Confirming, rescheduling and reminding patients about appointments and tests
• Answering questions about location, opening hours, documents and accepted insurance plans
• Sending test preparation instructions the team has already standardized
• Managing the waitlist and offering slots that open up
• Reconciling insurer payments and claim denials, flagging discrepancies for billing
• Tracking prior authorization requests and letting the patient know when they come through

Diagnosis, prescriptions and clinical guidance always stay with the health professional.

If a patient describes a symptom in the conversation, the agent doesn’t interpret it or give advice. It records it, says a professional will review it and, when there’s a sign of urgency agreed with the team, tells the patient to seek immediate care and hands the case to a person right away.

The technical safeguards that make a difference

Complying with the law depends a lot on how the agent is built. These safeguards tend to make the biggest difference:

• Least privilege. The scheduling agent sees the calendar and the patient registry, not the medical record. Each agent has its own account, with permissions only for its routine.
• Data in the organization’s environment. Data stays in accounts and cloud environments the organization controls, in the region its policy requires.
• Contracts with vendors. AI model providers need a written commitment not to use the data to train models and to delete it within defined periods, plus a business associate agreement where HIPAA applies.
• Everything logged. Every lookup of patient data is recorded: which agent, when, what information, for which case.
• Patient identification. Before sharing any information, the agent confirms it’s talking to the patient or an authorized representative.
• Lean messages. Reminders and notices don’t include a diagnosis, the name of a sensitive test or a result in the message text.
• Defined retention. Conversations and records are kept for as long as policy and regulation require, and deleted afterwards, including in logs and search indexes.

One point that often gets forgotten: content the patient sends, like a PDF of test results or a photo of a prescription, is sensitive data too. If it passes through the agent, it falls under the same access, logging and retention rules.

Clinics, hospitals and insurers: what changes

The principles are the same, but each type of organization has its own points of attention.

Clinics and private practices usually have lean teams and lots of conversations over WhatsApp, SMS or patient portals. The biggest gain is at the front desk. The main safeguard is keeping the agent’s channel separate, defining who can see the conversations and never mixing patient data with staff’s personal contacts.

Hospitals have more systems, more departments and more access profiles. Before switching the agent on, map which systems it touches and align with information security and the data protection officer. Administrative routines like authorizations, billing and test scheduling are usually good starting points.

Health insurers handle reimbursements, authorizations and member service at high volume. Here the rule against using health data for risk selection carries weight, as do regulators’ rules on response times and service. The agent can speed up document review and replies, with denial decisions always reviewed by people and clearly explained to the member.

What to ask before you hire

If you’re evaluating an AI agent for your organization, these questions help you tell whether the project was designed for sensitive data:

• Exactly which patient data does the agent access, and why?
• Where is the data stored, and whose name are the accounts under?
• Do the model providers use our data for training? Is that in the contract?
• How does the agent confirm the identity of the person on the other side?
• What does it do when a patient describes a symptom or an emergency?
• Is every access logged, and who can look at the logs?
• How does data deletion work when a patient asks for it?
• Who responds if something goes wrong, and how quickly?

Bring the organization’s data protection officer or privacy officer in from the first conversation. They’ll help define the legal basis for each routine, decide whether an impact assessment is needed and review vendor contracts. Leaving that conversation until the end usually delays the project more than any technical issue.