Turbopuffer Demo w. Davis Dunham

Hi, Davis.

Hey, Amber.

All right, so I saw your article about turbopuffer, for me and the audience here.

Can you tell us what it is in 30 seconds?

Yeah, absolutely.

So, we decided to upgrade our essentially, global.

Like our vault search.

So we keep an internal repo with all of our information and prior to this we were searching it using keyword search BM25, which works quite well.

I don't want to hate on BM25 but it was giving us the results that we were actually looking for about half the time.

So we switched to using semantic search which uses vector embeddings and actually encodes the like thematic meaning, the semantic meaning in the words so you can search.

The example I always give is funny.

it's like searching robot, vacuum and getting Roomba back, even though Roomba doesn't contain the words robot or vacuum.

Yeah, gotcha.

Okay, so you would say that the pain point this solves is not everything will say the exact words.

So keyword search is not always the best way to find what you want because humans express things in different ways.

Right.

So my question would be beyond from us, who is, who will benefit from this?

And how will this make their lives different?

I honestly think that most companies could benefit from this.

I think that as long as you have any amount of data that you're tracking and having to catalog, or organize in any way, shape or form, upgrading to a semantic search is slightly more costly because you're having to run it through an embedding model which is like essentially LLM, without its head almost.

so there is a small cost involved, but it's fairly low and it makes searching information so much more accurate.

it's like essentially the way I would compare it to the way I could provide an analogy for this would be it'd be like creating a Google search for your own internal documentation and data.

I see.

workflows.

What piece of the big puzzle does this solve?

Heard.

Yeah, absolutely.

So I would say what this solves is the grep problem.

So a lot of agents right now, especially coding harnesses and other like agentic systems, the way that they're finding information within like a mono repo or just any repo in general, is it literally just using keyword search and going through and trying to find relevant information.

Which is great, but it also requires multiple loops because the agent will search for something, not find it with the keywords.

It'll find similar keywords automatically and then search for those as well.

So generally these agents will find what you're looking for, but it'll take a while before it does.

Whereas this system will surface what exactly what you're looking for immediately.

for.

This is just skipping that whole step.

mcp.

So when you implement a turbo buffer, is it specific to One tool Or is this something that from now on all of the tools we develop and integrate will use?

So this is an exposed endpoint.

So we have this.

It consists of essentially three pieces.

And the three pieces are.

There's an indexer and the indexer runs on any new information or edited information that's coming into our knowledge, vault.

That indexer then takes let's say a document.

So this document is fairly long, a couple pages long.

What it'll do is take that document and split it into smaller chunks and then pass those chunks into a vector embeddings model.

And the embeddings model will then take like so you know, a couple sentences from this document and encode the meaning of those sentences into a vector and then store Those vectors on TurboPuffer in a namespace.

So that's kind of like the ingestion side of things.

And then you have the.

That's kind of fun.

The suggestion side of things which is like when you are querying it.

There is an endpoint that we have set up that runs the MCP server and essentially that just allows for any kind of.

It standardizes the way that any of our Systems interact with TurboPuffer because you can query TurboPuffer normally with like a rest, a REST request, like a post request.

But the MCP standardizes it in a way where we can now integrate this into any agentic system we have and it will work the same way across all of them.

So we have the deck creator.

The deck creator now is upgraded to use the Global Search mcp.

The Slack Assistant uses the Global Search MCP as well as trying to get our own coding harnesses to also use the Global Search mcp.

Okay.

So it takes from the sources and then you say what, what happens?

there.

So sort it takes from sources.

we have an indexer and the indexer.

Indexer.

Essentially it's just determining if it is a new or edited document.

goes.

And the indexer is in Turbo Puffer.

The indexer is something that we run.

Indexer is our own, our own code.

okay.

product.

But the thing is the issue with indexing is that it's like pretty specific to whatever format your own data is in.

So I, I, I actually didn't even look up if there were like already pre made products for indexing just because like I know brainforge does it a different way than most companies.

because there are different strategies.

But I think that it's pretty straightforward because it's literally just essentially scan all of the documents, determine if those documents are already represented.

Already represented in turbo buffer.

And if they are not, pass them to the vector embeddings model and then put them into triple buffer.

So if they are already represented, would it still go through this flow or no.

So it's, it'll skip them if they're already represented.

interesting.

Okay, So we have sources.

You have the indexer and then sends it to Turbo Puffer.

Yes.

And then what happens in Turbo Puffer?

cool.

So it's going back and forth to turbover to determine if the dog is already represented or not.

shape.

and then the model we used is.

Yeah.

Text embedding.3large.

Turbopuffer.

Does the vector embeddings?

No.

So vector embeddings are a separate API call to Azure.

Okay.

Though that Azure.

Azure specific is not important.

Okay.

prettier.

heard.

Okay, so it goes through the embedding tool and then.

Yes.

so.

And we store the embeddings into turbo buffer.

Yes.

vectorized.

Vectorized.

Entries.

docs.

Cool.

Vectorize docs.

Cool.

Now what happens after we have vectorized docs?

Then turbopuffer contains the information we need to query it semantically.

So it's a vector storage place.

Vector database.

So it's similar to say Vector db.

Yes, gotcha.

Okay.

Yeah, it's, they're, they're offering really what makes them different is the way that they manage their ram as far as I understand.

Because a lot of vector databases will charge you by like the RAM that you're using essentially.

So like how long you're using ram because they keep your name spaces, they keep your like your databases stored and ready for use like at all times.

And what turbover does is they just charge you by how much you are uploading and downloading, which is a novel way to do it.

And the way that they do, they do that is that whenever you have, whenever you haven't used your database for a second, they like spin it down so it's, it's not online anymore.

And then when you query it again, it has to come back online.

And then it can give you the answer, but it means it takes a little bit longer if you're using a system that needs information like immediately all the time.

Triple puffer isn't the way to go, but it's also way cheaper.

And if your system is cool with an extra like couple milliseconds of lag, then you use trip.

I see.

That's a very good take.

I will use that snippet.

So, okay, it stores vector embeddings.

And, and I think now we're going to talk about like once it's stored, the ingestion piece is done, right?

Yes.

Okay, so now we're talking about retrieval essentially.

So what happens between say the mcp server and turbo puffer?

What does the mcp server request and I guess what does turbo puffer give back?

MCP Server just manages the like API connection essentially.

So it's just.

MCP Server is just a way for the agent systems to It's like a, like a converter essentially for the agent can now send a request to the MCP server and the MCP server handles the formatting of how that request should look to the Turbo Puffer backend.

So I am going to say.

so I'll say request.

And then I guess the, the main difference then is how we search here.

absolutely.

So the vector embeddings are what enable the semantic search.

So the vector embeddings are what are storing the ideas beyond just like the keywords in the documents.

If you were to search you know, database problems.

That the only thing that would overlap there is data, but it'd be database.

So it still might not even surface exactly what you're looking for.

Whereas if you use semantic search and you search database problems, it knows that inefficiency and problem are related semantically like thematically.

Uhuh.

So, so you don't have to, you don't have to actually be as accurate in what you're searching.

So before did we just cut it up into chunks and didn't do embeddings so adding.

So yes, the vector embedding tool is the new step.

Yes, essentially.

And then the indexer as well.

well.

Because the indexer before.

The indexer before worked.

and it did do some modification of like, of edited files, that kind of thing.

So if something changed it would like, it would say something about that in the turbo hover namespace.

But now what we actually are doing is we're creating like a cryptographic Sha.

A Shaw, like key for all the files.

And so if the file changes a little bit it'll.

It'll now output a new shawl key.

modified.

And so I've been using this to our advantage in the indexer.

So one of the ways that it does this now instead of having to go through and actually look through every single bit of text in, in.

In in every document to, to look for modifications, it can just compare the, the shock key from the old document to the new document and see if they match.

If they match, it's the same document.

If it's changed, it's no longer the same document.

It's going to re.

Index it and then replace it.

Cool.

Actually I think it will help folks understand if we had like a.

Before to this.

like before?

Was this just stored in say S3 and then we just search S3 like you would do control command F on the Google Doc?

Yeah, yeah.

Essentially.

Yeah.

Wow.

Okay.

And it was still stored in TurboPuffer, but I don't know if we were really using TurboPuffer in any kind of centralized manner.

It was just like another copy of the repo, essentially.

I see.

So and so yeah, we do this.

It's essentially there still is an indexer but so we'll do sources.

I'm going to add this right here.

We still had Turbo Puffer.

But.

But then Turbo Puffer was pretty replaceable because we weren't really using anything else.

And I actually I'm going to do the repo separately because the repo was what all of the agent else's were directly requesting.

are.

And then there was like a very lightweight indexer that happened.

indexer.

Vectorization.

Cool.

Min.

Minimal modification awareness.

And then these also just went directly to the repo.

And there was an MCP server for this but it just didn't be in 25.

So this is going to do.

I'm going to add this right here actually.

Semantic search.

Did not mean to do that.

this one down here and over here we can do.

It's literally just grab.

It.

And then when we search via keywords it will only return exact matches.

Right.

And this, yeah this gave us a 59% success rate.

It.

So in the new, in the new thing the repo is also one of the sources, right?

yes.

tool?

why does having this vector embedding make it so much better?

but based off what I understand, the vector embeddings work similarly to the way an LLM understands language.

So when you're asking about, when you're asking Claude a question about something, the way that it understands the meaning behind that question and then can reason and respond to it is also the same way that the vector embeddings tool is will vectorizing the these ideas like these statements.

Right.

So essentially it's graphing them and then allowing you to do geometry on the ideas.

But it's like high dimensional geometry.

But like the actual in these tools are literally using cosine and sine and like dot products to compare how closely aligned these vectors are pointed.

I see.

So I would say that then why it makes search more accurate is because before say you have the thing cat, then you, you only have the three letters cat that you can keyword search for it.

But now you have all these dimensions like cat is an animal.

Cat is not dog.

Cat has these different shapes.

These are cats.

So when you, when you search for like feline that does not keyword match with cat, it still understands like overall these are the different dimensions to the meaning.

So like it gives characters meaning essentially.

Right?

Yes.

Okay, cool.

I think this was very helpful.

Absolutely.

That's exactly what's happening.

right.

in.

Do you want to screen share and kind of walk us through an example of how it works so we can see inside the tool.

Yeah, absolutely.

Okay, so take us to the repo.

Here we are.

Okay, y', all, I was going to demonstrate our turbopuffer, vector embedding, semantic search, global search, MCP today.

I know it's a long string of words, but essentially we're using turbo buffer to store all these vector embeddings.

And this enables our semantic search using our global search mcp, which is essentially an upgraded version of keyword search.

No longer.

We're no longer using BM25, for just keyword matching, like Control F on a document.

Now we're using semantic search which allows you to look up ideas that are related to, what you're searching.

So I have a claude session right here.

I am going to copy and paste this bit because I've set up a local BM25 keyword lookup in, this repo specifically.

And this is just the instructions for how to do that.

What we're looking for is the query is checklist to review before sending a statement of work.

As you can see, what we're actually looking for is this document sow checklist, which is just document that describes how to set up a statement of work for a client.

The BM25 comes up and pulls up, five documents at the top.

We have readme, sales documentation, SW, ads, onboarding, AI quality checks.

None of these are the document that I was looking for.

In fact, the document that I was looking for ranks number 18 not even in the top 10.

So even though the phrase that I looked up was fairly straightforward checklist review before sending a statement of work.

None of these documents are that document.

That's the downside of keyword search is that if nothing is exactly what you're looking up, you are not going to get the result that you were looking for.

I'm going to go ahead and start a fresh claude session, just so you know it's fresh.

And I'm going to copy and paste this exact same query checklist to review before sending a statement of work right here and not giving it any other, any other context.

And what we should find is that it pulls up the exact document that I'm looking for.

So this is a pretty quick and really good upgrade for any kind of agentic systems you're using that are interacting with your knowledge base or your repo or any vault that needs to be accessed.

This is a pretty huge upgrade over keyword search.

And all of our, evals have shown this so far, based on our golden data set.

Prior to this, our keyword search BM25 was hitting, I think 59% accuracy, which is honestly not great.

And we upgraded semantic search and we're above 90% now, so a pretty huge, pretty huge accuracy upgrade there.

There we go.

Look at that search.

Found the exact doc, sow checklist right there.

Exactly what I was looking for.

but.

Okay, yeah, show me inside Turbo Puffer so that people know what the tool looks like.

dial.

Here we are.

So this is triple cover.

It's, fairly straightforward in all honesty.

it kind of groups all of our information together.

So, the number of namespaces.

namespaces are vaults, essentially.

faults.

It's just individual, like portions of data.

So we have like a different tables of vectors.

Yeah, exactly.

We have a Slack namespace.

We have a couple different namespaces.

Cool.

And then we have rows written.

well.

and the namespaces themselves are pretty unassuming in all honesty.

There isn't a way to really look through them.

from this, from the space, it just shows us a list of the namespaces, how many documents and their size.

now that we looked at the benefits of this, like who would you say, who would you not recommend it to or what do you need to do before you jump into implementing this tool?

Okay.

I think that data, hygiene is important.

To be able to implement a system like this, you have to know what sources you're pulling from and those sources have to be up to date enough to be able to pull from them.

Like having a, like a global search utility or that functionality is, isn't very useful if there's a lot of information sitting kind of siloed places, or not being uploaded in the first place.

So you need to have the infrastructure set up with your organization already where that information is accessible and this is an upgrade to that accessibility and accuracy of accessibility.

I see.

So this is more of a advanced move to improve an existing working system.

But if they 1, don't have the data, or 2 the data is way too messy to even sort through or search from, then like this tool is too far of a jump for them.

Absolutely, absolutely.

You gotta, you gotta have your sources locked down and the data hygiene has to be good before you can implement any kind of vectorized database.

Gotcha.

Okay, let's see.

So another question I have is how would this differ from other tools and why pick this among the alternatives people have.

So in terms of Turbo, Puffer specifically, turbopuffer is a system that when you're not using it, when you haven't queried the database in a certain amount of time, it will actually put it in a bit of a sleep mode and they save money by doing this.

So the immediacy of a query does take a small hit, but we're talking about like half a second.

You know, this is, these are very, very short time span.

So if you're working with a system that needs information immediately, all the time with as, as little lag as possible.

Turbo Buffer might not be the vector database to go with, but you can't beat them on price.

So you can.

They charge, they charge by how much you're uploading and how much you are downloading from the database.

Which is a, as far as I understand, pretty novel setup for their pricing structure.

Yeah.

Anything else you would like to cover?

sorry.

I was gonna say I don't think that I have anything specific to add.

I think it's just a, I think that it's a a fairly easy upgrade once you have the data, hygiene sorted out and well worth it.

I mean in our testing we went from about, about a half, about 50.

It was a 59 success rate in our golden eval set to 91%.

So we nearly Doubled our accuracy in, in like single shot.

Searching our own knowledge base.

I see.

And also I, I hear another benefit is that once you do this, you can apply this global search to many of the different tools that you have.

So it's a like one and not one and done.

But you only have to do it once and you can apply it to different tools.

Yeah, absolutely.

And then once you've indexed, the hardest part is the first index because you have to re index all this stuff that already exists.

But once you've done that, that's the most expensive part in terms of like paying for the embeddings model.

after that it's just, you know, small modifications and files being added one by one.

So it can happen nightly.

It can happen even hourly if you want it to.

And if you have a robust indexer that's not going to re index stuff that's not necessary, that hasn't been modified, then you know, you're, you're getting, you're getting a huge benefit for pennies on the dollar in terms of how much you're spending on a system like this.

Gotcha.

Okay.

like that's the key change is that we're doing embeddings now.

But having a good indexer, having a good, economical vector storage is helpful.

Yes, absolutely.

Gotcha.

I couldn't say about myself.

Sight.