Building a Smart Glass Co-Pilot 👓

Hey, my name is Mika, and I built a smart glass co pilot on the Auki ecosystem for Mentra glasses.

And today, I'm gonna show you how I did that, and I'm gonna explain how you can build on top of it and do it yourself, so.

Okay, so the repo can be found here.

It's called ship memory the repo basically consists of two parts.

At one side you have the Mentra Live agent, which is a Mentra app for smart passes.

And on the other end, you have the Auki One shot bridge, which is an API layer on top of an Auki domain.

and using the one shot protocol, like the ship memory protocol, you can get context from the physical world into the smart glasses Live agent.

So you can basically talk with things that you see in the real world, as long as it's stored in a QR code.

So on the Akidomain, first, I want to show you how to set up the Mentra app.

So what we need is an, API key from the, from Mentra, and also a, package name.

For this, you first go to the Mentra developer portal, you create an account, and then you create a mini app.

A mini app is an app that works for smart glasses like Dementia Life, and you can download the Mentra app on your phone to open up any mini app.

So in the developer portal you create a new app.

you create a package name here.

This is the name that you need to use in the environment settings, the display name, the description and then logo.

This doesn't really matter.

later as well, you need to put a server URL here which tells the mini app where where the back end of the app is hosted.

So you can host it either locally with Ngrok on your own machine or you can host it in Railway.

I'm currently hosting it in Railway.

So here you can see where the app is running.

we'll get to this later.

For now, you only need the package name and the API key.

What's also important to add is required permissions because we're going to use a live agent, so we will need the cameras.

You can add a permission here.

You click permission type and then you basically pick camera, which is not shown anymore because we already added it.

And then you click done Done, which is important also.

What's important here is hardware requirements so that people with that don't have cameras cannot open the app.

Okay, so after you clone the repo and put it on locally, you will have to copy the env Example file and remove the example part.

Then in this file, you will have to set the port number where you want to host, the app, Mentra app, which is 3,000, I think, on start.

And then here at Mentra Client, you will add your package name, your Mentra package that you just added in the Mentra Dashboard, and also the Mentra OS API key.

And with these three things, so the package name, the API key, and the port number, you will be able to start up the Mentra app locally, So after you set the environment variables, you can start the app locally by opening the folder in the terminal, then run bundle and start client.

This will set up the BUN application for Mentra.

for me, it's running on Port 8000 and now you need to expose the app online so that you can find it in your Mentra app, on your phone.

We, can use Ngrok for this.

So this will export my port 8000.

It's connecting now.

And then you take this URL, copy it, and then you add it here, server URL.

This is important because then when you open the Mentra app on your phone, it will know where to look for and which app to start, basically.

So right now, if you set everything up correctly, you can go on your phone, you can start up the app.

And what happens now is that the application starts setting up a WebRTC stream to your own application.

And a QR code resolver reads, not every frame, but a lot of frames and checks if there is a QR code visible.

It doesn't set up a Gemini session just yet.

It just waits for a QR code to be, detected, basically using the protocol, the SHIP memory protocol that you can also find in this repo.

this is how it works.

It knows how to get the correct information from the QR code.

So for the next part, going to show you how you can create context that you can view with a QR code on, top of the Auki bridge.

Okay, I forgot one thing to add.

as you can see in the environment file, there's also a Gemini API key that you also need to set.

You can get one here.

So you go to Google Gemini API or Google API.

And then I think here, with API keys, you log in with your Google account and then create or view a Gemini API key.

And then you can get, I think a free API key is enough.

up.

No, I think you need to set up Billing.

Set up a key here, copy it, and then you paste it in the environment settings.

And now the Mentra app is ready completely, to set up a Gemini live session.

Okay.

Now we're going to dive in, to Aoki Bridge.

To set up the Auki bridge you first need an Auki domain.

And for an Auki domain you first need to create an Auki, login.

So you create an account here, Auki network console.

This is the URL.

You sign in, Create an account or sign in.

And then here on the left side, you can build your own domain.

You get a wallet assigned basically with I think a limited amount of tokens to already be able to set up a domain.

I have three here.

You can create a new one here and when you created one you will need to get the ID here.

So copy that then back to our application in the environment settings.

Here you have to log in, you don't need to log in here, but if you're developing, it's easy to do so so you put your email, your password and also the domain that we're going to use.

So in this case I just copied it, you add it here and it will work.

Now we're going to set up the Auki bridge locally.

You go to the main route, open a terminal and here you type Bun run dev which will start the Auki Bridge locally.

it sometimes gives an error if it's on the same port as the Mentra app so you need different ports for this and you can now see that the bridge is unconfigured and you can complete the setup at the UI endpoint which I made.

So you can also keep all the login environment settings empty.

And then when you go to the dashboard so your localhost port exposes UI or if you host it on Railway, which I will explain later, your railway URL UI and you'll be able to see this.

so here's another, another instructions on how to set up your Auki Etc Auki domain.

so if you got it you can press continue So we got the domain here and for me this will show already stuff that I added for you.

It will be empty.

It will now mint a token.

So the bridge locks into the Auki domain.

And this is why you also need an API key.

Because your client apps on Mentra Mentra don't have to log into Auki because they can just use an API key to get access to a domain which makes everything easier.

you can add whatever you want here.

The most important thing to do is if you set up an API key you have to set it in your environment settings as well.

So it's here Bridge API key.

You can also change it later.

So it's quite easy.

But it's important that each client has this key because otherwise they will not be able to fetch information on the QR codes.

Okay, so after you logged in, it will save your login as well to the environment settings, so you don't have to log in anymore.

You just enter your API key to get access.

and here, so now you can see the dashboard.

In here you can deploy a new card with an id, like a name and the body.

The body here is the context prompt.

So here you can put a large markdown file that has instructions about anything.

Basically, maybe you want to have an AI agent that helps you with making pizza.

You can add recipes here.

Then you click deploy.

And this will save it to the Auki domain itself that will look like this.

So here you can see, that a QR code will be generated.

And this QR code is linked to the context card that you just made.

And when someone scans this QR code, they will get that information.

So those are stored on the akudomain.

Another thing that you can do, with this protocol is that you can use tools for the AI to get more context or to help you better.

And these tools are kind of embedded into the way you present the context to the live AI agent.

this is what I think is really cool.

So I made one tool example, and it's session history.

So in the context card, there's also a tool description and it says, hey, if anyone wants to save information about this specific context, like, okay, I want my pizza without pineapple because I hate pineapple, then the AI knows that it has to save it.

And it can do so because there's a tool available for this.

you can add more tools yourself, for instance, to get live information or to get more context from somewhere else that's also on the domain.

You can even put a script in there.

Whatever you want, you can build it on top of it.

And the beauty of this is that the protocol, which you can read, will be still the same.

So we can all build on top of it.

That's the idea.

And the client apps don't really have to do a lot of things because everything is embedded in the protocol and the API bridge itself.

what did I forget?

this is now running locally, but as you can see here, this URL is Auki Bridge production.

It's on Railway.

On Railway, you can host applications, which makes it much easier for you.

So to do this, you create a Railway account.

I think it's €5amonth.

I'm not sure.

Maybe there's also a free version here.

You can create, environments like sandboxes, and you can deploy Bun applications with all kind of applications.

So you can run your Mentra app here and also the Bridge app, so it's always online.

so here you add a new service, you press from GitHub, repo, etc.

What is very important is that in the Mentra console you will add the server URL here because it's not longer running locally, it's running on railway.

So you have to add this URL code.

but now it should work.

So when you scan the QR code, it will get the context that you just described and a Gemini session will start up using that prompt.

This is good and all.

but what if you already have an Auki domain and you're already using QR codes?

So to use existing QR codes you have to copy this link.

So this is the link to our deployed QR code.

then you go to your Auki console, which, your domain that you just created and go to Portals.

you can view it here with this button and in this portal tab you can add QR codes, redirect.

So you create a new one with this button, add a name and then put in the redirect URL, which is the, the one you copied.

But you will have to add your own API key as well.

the API key that you created.

Now when you scan this QR code, it, you can create one here with this button.

What is also important, you will have to whitelist URL codes But how it works right now is there's like specific URL prefixes that you can whitelist.

So you will not be scanning any trash or anything.

Okay, you go to the environment settings.

For me, this is in real way.

You go to your client because you have the whitelist per client.

You go to variables and allowed URL prefixes.

So, and then you can change it.

I don't know how to change it here.

So you have to press RAW editor.

I won't click on it because then you will see all my API keys.

So but you can do it yourself.

So you whitelist the URL codes basically.

So this is how you can use the Ship memory Auki bridge.

There are things to improve.

feel free to, to add to it.

Feel free to send me a message on how it works.

I'm always available on Discord.

I'm in the Auki, Discord as well.

So you can just send me a message.

yeah, I'm curious what people will build on top of it.

And I think this new way is pretty interesting because you can now walk around with smart glasses and you can work on things, that you've never seen before or you can get context from your environment, which cleanly separates the context from the AI agent, which I think is the future.

so thank you for checking this out.

yeah, that's it.