Embedding Corti Assistant: First Working Integration

Hi, I'm Morten, staff engineer here at Corti and I'm working in developer experience.

In this video I'll show you how to embed Corti Assistant into an existing clinical workflow.

The key phrase here is existing clinical workflow.

All clinicians already have somewhere they an ehr, a visit screen or some other clinical application.

The app already knows the patient, the encounter, how to fill the documentation form and what happens to data.

We're not trying to replace that workflow, we're putting Corti's clinical assistant inside of it.

What I show here is a completely fictional patient system and I'll use that throughout the series.

It's deliberately simple but it has all the pieces we care about, it has patients, it has patient context, it has different encounters and it has a form with fields that we need to fill.

At the end of this video we will have Assistant running inside of this EHR automatically filling all of these fields for us.

In this video we'll kick off looking at the integration model.

This explains what the host owns.

This explains what the assistant owns and in particular explains how the two apps talk to each other.

Then we get to something useful.

We'll mount Assistant inside of our EHR.

We'll configure it for a narrow strict use case.

We'll record a conversation, generate documentation, and we'll finally sync that documentation back back into the EHR workflow.

In future videos we could get into some of the messier real world cases, mapping unknown output into existing EHR structures using dynamic inline templates, multi user setups, brokered authentication, and the production concerns around some of those things.

Embedded Assistant is best utilized where there's already a clinician facing app.

I'll call that the host application.

That is often a desktop EHR application, but it could also be a visit screen, an app on a tablet, or any other documentation workflow.

This host is the driver.

It owns the surrounding patient and encounter context and the overall workflow assistant simply needs to fit inside of it.

If all you need is back end processing with no clinician facing interaction, for example just processing recordings into documentation, we have other Corti APIs that are much better fit.

Embedding Assistant is used specifically to bring the assistant UI and experience into the workflow that the clinician already uses.

One quick implementation detail before we get any further, how you embed Assistant depends on whether the host is web based or a native application.

A web app could be JavaScript or TypeScript, React, Vue, Angular.

this also includes wrapped web apps like Electron which are popular for cross platform native like web deployments.

In web based integrations, the host application renders the assistant web component directly for a native app could be .NET C# Swift on iOS or Kotlin on Android.

A go between Webview is needed that webview loads a small local HTML page managed by the host application and that page then contains the assistant web component.

Communication here goes from the native app through the webview and the local HTML page to the web component.

The outer shell is different, but both routes end up at the same web component boundary.

The main difference is how complicated the bridge between the host and assistant is.

This demo uses the webpath.

It's the next year's web app, so we'll talk to the component directly from TypeScript.

However, we are also working on equivalent native examples including a C# .NET example.

One important thing to get right is the boundary between the host app and assistant on the host site.

The EHR owns the user and encounter context.

The EHR controls the overall layout of the application.

It also determines the workflow constraints, which templates are available, which languages are allowed, and so on.

The EHR also manages the form data and how the assistant output fits into that.

And finally, the EHR owns the local state and preferences.

Assistants only takes care of what happens inside the embedded clinical experience.

This includes the microphone lifecycle and audio capture, the real time API and socket communication happening behind the interaction, including some pretty complicated error handling.

It handles template selection and the generation of documentation.

And finally, it exposes the events and methods that enables the communication with the host.

Communication goes both ways.

The host sends context and configuration into Assistant authentication, where the interaction belongs, which templates are available, how the experience should behave.

Assistant sends things back, events based on state changes and generated output.

Before we touch the code, here's where we're headed.

What you see here is the assistant embedded directly inside the mock EHR.

The clinician stays in the same app, the patient and encounter constant stays on screen, and Assistant sits right there alongside of them.

We're about to record an encounter with Eleanor Shaw for an annual checkup, and it might go something like.

Hi, Eleanor.

How have you been since your last visit?

Pretty well.

Overall, I've been a little more tired than usual, but nothing major.

Any new symptoms or concerns?

I'd like to talk about whether I should be exercising more.

I mostly just walk a couple of times a week.

Okay.

Your blood pressure today is 128/78, and your weight is stable.

We'll continue your current medication, arrange routine blood tests, and I'd recommend gradually increasing your exercise to around three or four times a week.

Sounds good.

Then we stop the interaction and we let the note generation complete and we can review those notes for accuracy.

But for now, we'll just go ahead, generate the note, and once the note is generated, we can review it, and if we feel happy about it, we can synchronize the document back to the ehr.

When we press this button, we hide the assistant and we can see that the information is now written directly in the correct field in our EHR form.

This is the smallest useful version of an embedded assistant integration.

Before we zoom into the code, let's take a look at this flowchart that explains the entire data flow in a single diagram.

The host embeds the system and waits for the onReady event.

Once we have that, we can authenticate.

We can apply whatever app and interaction configuration we need.

Create the interaction, we optionally add some patient facts and we navigate to the interaction.

With that done, we show the assistant UI.

This all happens within a few seconds at most.

From there, the clinical work happens inside of Assistant.

Recording the patient interaction, reviewing facts, choosing a template, generating the document, making a few edits.

This might take minutes or even hours.

Neither the clinician nor the patient needs to look at the screen while recording takes place.

Notes are captured automatically for the clinician's review after the interaction, When the clinician syncs the information back to the EHR, Assistant sends a document.synced event back to the host.

The EHR maps that output into its own fields in the form, and the workflow can also hide Assistant again afterwards if necessary.

The host handles the setup and the form filling, and Assistant handles the clinical workflow in the middle.

For this first implementation, I'm deliberately cutting down the number of moving parts.

I want the smallest example that still shows the full integration clearly.

So although this EHR has several encounter types, I'm only enabling Assistant for annual checkups.

Everything else stays exactly as it was.

For the same reason, I'm also fixing the documentation to only one SOAP template.

That gives us a predictable structure.

We know where subjective objective assessment and plan should land in the form I'm keeping a couple of other things narrow too.

English only, explicit sync and a single user setup.

A real integration usually has to map the signed in clinician to a Corti user.

We'll come back to that in a future video, including brokered authentication.

This gives us a small predictable first example.

Once it works end to end, you can start adding flexibility and expanding it one piece at a time.

This narrowing decision also shows up in the code in a very clear way.

In the existing patient interaction the normal EHR foam is still there.

We have patient information up here, we have the consultation form down here, and right here.

If consultation type is annual checkup, we simply render Corti Assistant.

That component goes through a few layers of UI, but it ends up at this component here, where we embed the actual Corti Embedded React component which is loaded from the @corti/embedded-web package which is live on npm.

We pass in some properties, we wire up some events, we initialize it to hidden while we set things up, and that's it That's enough to mount Assistant.

From here, Assistant owns the clinical experience inside of that container.

is.

Assistant is now part of this EHR workflow.

But mounting is only half the job.

We still need to initialize it for this particular visit.

This function does that setup through a sequence of steps.

First we authenticate, then we configure the app.

We set interaction options, then we create an interaction, add some facts related to the patient.

Finally, we navigate to the session we just created.

After that setup is complete, we can now show the actual UI to the clinician.

The important part here is, of course, which interaction options we pass in.

And here you can see that we limit the spoken language to English, we limit the output language to English.

And we set a forced first document, which we've hard coded to the SOAP template id, which is a uuid.

We also enable the synchronization button, And we limit the number of documents allowed to only one.

Note that these aren't Corti defaults.

These are the settings that make sense for the flow that I'm creating in this application.

Your flow is probably different from this, so you'll have different settings here.

There's a lot more you can configure than what I'm showing here.

I'm not going to go through all of the options, but if you go to our online documentation, you can go to the configuration scenarios and you can see tons of ways you can configure the application to what exactly you need to display and how you want the experience to be for your users.

The actual API I'm using in this demo might evolve over time, so always check our online docs to make sure you're using the correct API.

The other side of the integration is handling the document once the clinician is done refining it.

When the document.synced event arrives from Assistant, the host receives the generated sections.

With a fixed SOAP template, the mapping is almost boring.

We simply map the subjective field to the subjective field in our form, the objective section to the objective field in our form, and so on.

this code is just that mapping.

It tells the host which form field each section belongs to.

So sync can fill the form.

Here, Assistant has generated the note, but nothing has changed in the EHR yet.

The form fields are still empty.

That's intentional.

Document generation and EHR form filling are two separate steps.

When I click Sync, the host will receive the document and map those known sections into the known fields.

We also collapse embedded assistant to save space.

It's no longer needed.

Now the documentation is back in the EHR without the clinician ever leaving the host app.

At this point, we already have something useful, we have the assistant embedded, we can configure the session, we can use a fixed template, and we can pass in patient facts.

The user can generate a note, they can review and sync that note, and finally the EHR can fill the form.

Before this is production ready, we need to add some or all of the stuff you'd expect around this.

Authentication and token handling, loading states, error handling, retries, permissions, telemetry audit and logging.

This demo app already has some of that.

I just didn't go through all the details and it's definitely still lacking some hardening before it can be considered production ready.

This first mapping was easy because we cheated a bit.

We locked the clinician to just one SOAP template.

That's great for this demo, and it might even be sufficient for some integrations.

But it does sacrifice a lot of Assistant flexibility.

Assistant offers a ton more capabilities than what we have here, including a large library of standard templates across many specialties and the ability for clinicians to create their own templates either from scratch or by modifying a built in template.

If we wanted to allow clinicians a choice of multiple standard templates, we could hard code mappings for some subset of those.

It would work, but it would quickly become a lot of manual lookup work.

However, with personal templates, things fall apart.

The host can't know every possible section ahead of time, so eventually the host will receive a section and doesn't know where to put it in the EHR form.

At that point, the hard coding mapping stops being a viable solution.

We'll show you different ways to solve this in future videos.

So that's the first two pieces done.

We've discussed the integration model and we have a working first embed.

From here we'll use this same demo to push into the messier real world bits in future videos.

More flexible templates and mapping, multi user setups, authentication, other production concerns.

That's it for this one.

You can grab the example and docs from the links on screen or in the description.

I'll see you in the next one.