📱Intro: Companion App Setup & UX (Public)

Hi.

So recently in the Assistant Embedded experience we introduced a few new things related to your microphone quality.

For example this sound wave which shows your live signal for your microphone.

Also a set of callouts which tell you if you have an acceptable suboptimal or optimal microphone as well as the ability to set your ambient or dictation default microphone.

So all of these new additions stem from the fact that we want to help users use the best possible microphone.

so what we found out is that oftentimes you users don't have a good microphone setup.

So they either need to have their organization invest in, such setup or we thought what if we could utilize a device that everyone carries in their pockets as a microphone input device.

That's where the project came along to allow users to pair their phone as a companion so that they can use their phone as a microphone input source.

So starting now any embedded assistant integrator can enable this functionality for their users and configure it.

So if I'm in the shoes of an integrator, what would I do?

I would go to docs.corti.ai and I'll find the Corti Assistant tab and go to Configuration Scenarios.

And now there's a new section here which says "I want to allow clinicians to pair the Corti mobile companion app" and then I'll see what kind of prerequisites do I need.

So I can see that I through the API I can configure the app with the companion app enabled.

And then I should pass my access token, refresh token and token type.

In most integrators case, the authentication or access token is already in place.

So there is not really any step there.

so what I should really do is I should configure how to show the device linking QR code to the end user.

So to do that I will go in that page down here, show device link QR which will Again show me the usage, how to configure it, but also a bit go a bit more in detail on the prerequisites, the input validation, what types of returns I might get, and also possible errors.

Now to demonstrate how this workflow works, I'm going to use the Corti Embedded playground here which looks like this, which on the left hand side allows me to control a whole bunch of API controls from authentication to configuring the app.

So what we'll do here is simulate what an integrator would, would go through in code just with a slightly more user friendly ui.

So to start with, if you remember we have to first set that the companion app is allowed.

To do that, we'll go down here to configure app and enable the companion app.

Now that we've done that, then we can move to authenticating and linking the device.

This, that is the first two options up here.

So I'm just going to, I'm actually authenticated here, but I'm going to authenticate again just for the sake of this example.

And then I'm going to add my access token, refresh token and token type in here as well.

So then get to show the device link QR and there it is.

So the idea is that this URL lives in a separate URL which is an integrator, I can choose how to present to the user or where should that live.

one suggestion here is that this is meant to live for example in the EHR example in your EHR settings where, where you might have a section called pair your mobile device here and then triggering a button there which display this UI that we're looking at here, where as an end user I can take my phone, I can scan this QR code and then I can use it as a external microphone.

So to demonstrate that I'm going to do exactly that, I'm going to take my phone and share my screen and walk you through the steps.

All right, so now I have the app installed.

I have opened the app.

You can also see it on the right hand side of the screen.

so what I'll do is I will just click scan code from the desktop.

And then when I do that I can scan the QR code and you can see that I'm asked to confirm or deny the connection.

So the end user would see this, on their screen and then they can they can check which device is trying to connect, in this case my iPhone 13 Pro, and And then now I'm fully signed into the app and the app itself is paired.

I will actually also get a success message here, which is not displayed in this prototype, that your phone is now paired successfully.

So that now that I have the app paired and I have for example an interaction, next time I have a patient, I can go and navigate just here to my current interaction.

Say I have a patient coming in.

So this is the familiar view that we know, from today.

in this case I have the app open.

I need to have the app open in order for this to work.

That's step one of the workflow.

so when the app is open, it's paired.

You can see that it says that it's waiting for a session and you should go as a user in the desktop, on the desktop and choose your phone as your input microphone device.

So I'll just do that.

so then if I click here to microphone source, I can see my phone is listed as an option that I can choose it.

And now you can see that, my phone also reacted to it.

So I can pick it up and either from the phone or from the desktop I can say record.

I'll just do it from the phone.

And as you can see the language of recording is being carried through as well and we're recording.

We can also see an indicator, recording indicator on the phone side of things, as well as the option to pause.

same thing as a desktop, which we already know how it works even today.

we can also see that transcript in fact comes through here and on the phone, a little, addition is that you can also see the instant transcript.

So as I'm speaking, bits of transcript appear to let me know as a user that the recording is in fact happening along with of course the time elapsed and the indicator that is recording.

We also have while recording, a reassurance indicator here that you're connected to the desktop.

So once your consultation is over, or as a user, once my consultation, is over, I can pause either from the phone or from the desktop.

so I'll just do it from the phone in this case.

So now we can see our session is paused.

so if I want to, I can continue from desktop or I can also continue from the phone directly.

Both work.

so then also in settings here, I can see that this microphone is still selected.

and I can also look at the transcript right, from there, if I choose to continue, I can do that again.

And then the session picks up a new recording, appended to the previous one.

Same as how it works in the desktop app today.

So I'll just hit pause here.

so really with this, new functionality, you can place your mobile phone between your patient and yourself as a doctor or as a clinician or wherever you think that you will get the best kind of microphone, input or voice input in order to generate, a document of good quality.

So then once I'm done with this patient, as I can move to my next patient.

So in my EHR, I would open up the next, patient interaction that would create a new interaction.

I can just simulate that here and then I will navigate to it so you can see what happens on the phone is that now it resets to the awaiting session screen.

Where I can move to selecting my phone as an input, so that I can start recording again.

So one common use case we want to handle with this flow is also what happens when your phone locks by itself.

So let's say we have a recording here.

I'll just start it.

And as you see, we're ready to record on the phone side.

It's initiating and the recording is now ongoing.

So it's elapsing.

But then if I take my phone and lock it or out it auto locks.

You can see that actually the recording here carries through or it continues.

the recording itself also transcript will be coming, in here in a few seconds.

but if I go back and unlock my phone, you could see that nothing has been interrupted, disturbed.

my, transcript is captured.

If I open it up, it should be here now.

and the recording, is now, continuing as it should without any interruptions from auto phone locking.

That's quite a common, way to disturb an application like this.

So just want to point that out.

One other thing here is as you're recording, we purposely made it so that you can't actually change the microphone so that you would need to pause to choose a different microphone.

So source.

But if you wanted to, you can do exactly that.

So if I just pause here, I can say, use my MacBook microphone and then I can continue recording with a different mic, depending on my use, case.

Right.

So that is another interesting little, detail here.

So the last thing I want to show you here is say as an integrator you can choose at any point to remove the companion mode capability if you choose to.

So you can go and say from true to false for the companion mode.

so then the end user, what they would see is well they have the phone paired so it will still be there.

But until you re enable that option then the phone will be disabled.

also say for in the example you have it enabled, and you want to record virtual session, then that phone is again disabled because for this mode it's likely you would have a headset or special kind of device to record for.

So we've purposely recorded remove that option from the list here.

Of course, if you disable virtual mode, then you can still pick up your phone as an input source as you typically would.

And lastly, if you happen to forget to open up the app before you start a session, I'm just going to close the app here.

You'll see that your phone here is, can't be found.

So again we have added guidance that that you should have the app open, when you want to use your phone as a microphone.

So you can start the app.

And then already again we are waiting for a session and the app page picks up on the desktop that the phone is available so you can start your recording.

That's been it.

That's been the workflow.

We're super excited to hear more of your thoughts, impressions and feedback.

So thanks for watching and talk to you soon.

Bye.