Regulation by design: designing compliant AI health tools

How is General Purpose AI in healthcare being regulated?

As tech giants continue to advance their AI health tools, new regulations are emerging to ensure these remain carefully classified and regulated. A key development in this area was the release of the updated Clinical Decision Support (CDS) Software Guidance . In the new guidance, the FDA has cleared the fog surrounding AI-powered clinical decision support tools, clarifying which digital health tools are lower-risk versus those under regulations.

For health-tech innovators, the era of permissive ‘grey areas’ is over. The traditional Silicon Valley ethos of ‘move fast and break things’ is no longer a viable strategy for clinical software. Today, success requires adopting ‘regulation by design’ from day one. Developers must ensure that legal and clinical guardrails are embedded in code not as a post-launch checklist, similar to how ‘privacy by design’ became the standard for GDPR compliance.

What is regulation by design?

Regulation by design adapts the principles of privacy by design to AI integration in healthcare. It means building compliance into the technical and product roadmap from the start, designing the model, UX, prompts, oversight mechanisms and claims the product makes so that they all align with the intended regulatory pathway. In practice, it requires teams to understand the regulatory classification they are targeting and to shape the system so that it operates and innovates safely within those boundaries.

The regulatory pathway for AI-powered health tools is hugely dependent on understanding the user, i.e. patient versus healthcare practitioner (HCP). The question is, how do you design compliant AI health tools for these two different user groups? Answering this question is key to the approach of regulation by design.

Developing “non-device” medical AI software for use by HCPs

If your solution targets clinicians, there is a distinct ‘safe harbour’ to innovate within. Under Section 520(o)(1)(E) of the FD&C Act, your software can be classified as a “Non-Device”, meaning it escapes the same rigorous and costly regulatory process required for medical devices. However, this is not a loophole for developers, it is a narrow gate defined by four strict criteria.

To qualify as “Non-Device Clinical Decision Support” under the 2026 guidance, your architecture must meet the following technical constraints.

No raw signals (Criteria 1 & 2)

Your pipeline cannot ingest or analyse medical images or raw signals (like ECG waveforms). Only discrete medical information (e.g., text-based lab results) can be used as input.

Recommendation, not directive (Criterion 3)

The LLM output must be framed as a suggestion. The UI must be designed so the HCP can “review, revise, and finalise” the output – a ‘human’ perspective is still necessary. If the software “replaces” the doctor’s judgment, it is a medical device. This principle is outlined by Human-in-the-Loop (HITL) architecture – integrating meaningful human review at the point where recommendations are generated and acted upon.

The transparency mandate (Criterion 4)

This is most challenging for LLMs. The HCP must be able to independently and autonomously review the basis for the recommendation without significant automation bias. One strategy is using Retrieval-Augmented Generation (RAG) to cite specific, verifiable clinical guidelines or peer-reviewed studies for every claim made. There must be sufficient time for the HCP to do this. A system that satisfies criterion 4 in one setting, might not satisfy it in a time-critical context.

If you satisfy all four criteria, your tool qualifies as a Non‑Device CDS function. Without triggering the full medical device regulatory process, yet still supporting clinical decision‑making, teams that intentionally design for this pathway gain a significant speed advantage, while still delivering safe, clinically meaningful functionality.

Navigating the three-tiered regulatory pathway for patient support tools: risk-based enforcement

If your General Purpose AI tool talks directly to patients or caregivers, regulatory guidance is less forgiving. The FDA considers software that provides recommendations directly to patients a medical device by default.

Unlike the single framework for CDS tools, this space is governed by a combination of primary guidance frameworks. The most relevant documents here are the Policy for Device Software Functions and Mobile Medical Applications (2022) and the General Wellness: Policy for Low Risk Devices (2026), released at the same time as the CDS update.

This is where strategic design comes in. The FDA applies “risk-based enforcement”, a tiered approach based on the intended use of the software and the potential risk to the user if the software fails. Regulation by design in this context means intentionally shaping the product’s functionality, claims and user experience so that it aligns with the most appropriate regulatory pathway. By clearly defining the tool’s intended use, boundaries and risk controls, developers can position patient facing AI features within lower risk enforcement tiers where appropriate.

What are the FDA risk-based enforcement tiers?

Tier 1 – Software functions that are not medical devices

This tier includes software that does not meet the legal definition of a medical device under the FD&C Act. In the context of ChatGPT, this typically includes “general wellness” tools. These are applications intended solely for promoting a healthy lifestyle such as a fitness coach or a sleep hygiene assistant, that are unrelated to the diagnosis or treatment of a specific disease.

Tier 2 – Software functions under enforcement discretion

This is the relatively ‘safe harbour’ for tools that technically meet the definition of a medical device but pose such a low risk that the FDA does not intend to enforce regulatory requirements. Many of the most popular health-tech uses for ChatGPT fall within this. These include tools that help patients track symptoms or manage a chronic condition without providing specific treatment directives, or “that are specifically marketed to help patients communicate with health care professionals”.

Tier 3 – The focus of regulatory oversight

Software in this tier is the primary focus of FDA enforcement because its functionality could pose a direct risk to patient safety if it fails to perform as intended. If a GPAI-based tool provides a specific diagnosis or directs a patient to take a particular medical action (like adjusting an insulin dose), it moves into this tier and requires active FDA clearance. This requires careful consideration of the core principles of AI design and development.

Applying regulation by design: staying in enforcement discretion

These strategies can help aim a tool for the lower tiers:

• Self-management and coaching: Design the prompt engineering to focus on “coaching” and “adherence” (e.g., “Remember to take your meds”) rather than “diagnosing” (e.g., “Those spots look like shingles”). Strict guardrails should block medical suggestions.
• Reference material logic: Treat the LLM as a “Digital Medical Dictionary.” If it strictly provides educational info already available in the public domain, it is much more likely to remain under enforcement discretion.
• Sensor guardrails: For wearable integrations, ensure the logic provides only wellness insights. Under the 2026 update, if a sensor prompts a specific clinical action, such as telling a user to “Take a new medicine” based on a heart rate reading, it immediately triggers active regulatory oversight.

The 2026 sensor update: Whoop and wearables

A significant addition in the 2026 guidance is the clarification on non-invasive sensors. This guidance is specific to software integrating wearable data, like the latest blood pressure estimation tech from companies like Whoop, who initially triggered increased FDA scrutiny as they ventured into a regulatory grey area. The updated guidance now draws a firm line: sensor‑driven AI remains exempt only when the insights are strictly wellness‑oriented and do not direct users toward any immediate clinical decisions.

For developers, this creates a clearly defined “innovation lane” for wellness‑focused GPAI products, but it also raises the bar. Systems must include robust guardrails to ensure they never cross into clinical recommendations, even inadvertently.

Conclusion

Building an AI-powered software product that operates in the healthcare and wellness space without first defining its regulatory status is a costly mistake.

For HCP-focused solutions, aligning the product roadmap with the FDA’s four criteria is an effective way to pursue ‘Non-Device’ status. Prioritising explainability and maintaining a Human-in-the-Loop architecture allows these tools to support clinicians while remaining outside the scope of active device regulation.

For patient-facing tools, positioning the software within the bounds of enforcement discretion is a common strategic choice. Focusing the feature set on low-risk support, rather than diagnostic or treatment logic, allows for a more streamlined path to market.

If your vision requires the product to provide specific treatment recommendations to patients, plan for active FDA oversight from the start. While this route requires a comprehensive commitment to clinical validation, a regulatory by design approach ensures that the resulting innovation is built on a foundation of trust, ready to meet the standards of the world’s most rigorous health authorities.