In v1.17 your Bunny changes how it looks for an answer. It walks your database the way the database is built to be walked: tables first, then the indexes and tags of the tables that fit, then the records themselves, and when it does not know the exact spelling of a name it searches by the beginning and by the ending of the words you used. Spreadsheet figures now come from the indexed rows, the same rows every other question is answered from. The result you will notice: a question that spans many files comes back in a handful of steps instead of one trip per file. And the other end of the pipeline got faster as well: indexing a spreadsheet now writes its rows about three times as fast, and up to three files are indexed side by side.
Why
We measured it on a real question. A project holds one settlement workbook per month, thirty-six of them, and the owner asked for one artist's distribution revenue across all of them. The answer was right, with the row count and the sheet it came from. It took 108 seconds.
The time went into the route, not the data. Until now a spreadsheet figure took a route of its own: your Bunny asked the server to read the workbook itself and compute the number, once per file. Each of those reads is fast on our side, about half a second, but each one is a full round trip through your AI platform, and the platform runs them one after another. Thirty-six files, forty-one trips, ninety seconds of waiting for a sum the database already held.
It held it because that is what indexing is for. When your Bunny indexed those workbooks, every row of every sheet became a record, with the artist's name on it as a tag. "Everything about this artist, in every monthly file" is one query over one table. So from v1.17 that is what your Bunny asks.
How your Bunny searches now
The procedure is the same for every kind of file, a spreadsheet, a PDF, a contract, an e-mail.
- Tables first. It lists the project's tables, each with its record count per access group. When the list runs past one page, or no table name obviously fits your question, it searches the table names with the words you used, by their beginning and by their ending, and keeps every table that fits rather than the first one.
- Then the names inside each table. For each table that could hold the answer it reads the index names, which come with counts, sums and averages, and the tag values, which come with counts. When the exact spelling is unknown, it searches those both ways too: "starts with" and "ends with".
- Then the records. It queries the table with whatever matched: an index value, again by its beginning or its ending when it is text; one exact tag; or the whole table. Always to the end. A single page is a sample, and your Bunny does not count, total or declare something absent from a sample.
- A known id skips the walk. A file name or a record id, whole or in part, is looked up by its beginning or its ending across every table at once, and a file's full set of records is one call by reference.
The exact file names, tag values and sheet names it used stay in the answer, so you can check them.
What a spreadsheet row looks like in your database
Each row of each sheet is one record. It carries the cells under their header names, and four fields of its own: the sheet it came from, its row number, its grain (a detail row, or a row of a sheet that re-totals another), and, for total and heading rows, its row type. Its tags carry the file name, the sheet name, the grain and up to four key values of the row, a name, a code, a category. A record also points back to the file it came from.
That is why a figure can be computed from the rows and still be exact. Workbooks routinely state the same money more than once: a detail sheet, then per-song, per-album and per-artist sheets that each re-total it, plus a summary sheet whose bottom row is the file total. Your Bunny uses the detail rows only, leaves the total rows out, picks one sheet per file and says which, fetches the complete matching set, and adds it up in groups, checking the groups against the whole. The answer names the row count, the sheet and the files it was made from, and marks the figure as calculated, so you can tell what the data says from what was worked out.
What to expect
- Shorter waits on questions that span files. The per-file loop is gone. What remains is the walk above, a few steps whatever the number of files, and the answer streams as soon as it starts.
- One route for every question. The spreadsheet tool that read whole files per question is no longer offered to the chat. The indexer keeps it: after writing a sheet's rows it still asks the file for its totals to check its own work, which is how a sheet gets verified before it reaches you.
- Nothing to re-index. Your rows already carry the sheet, row, grain and tag fields. The change is in how your Bunny asks, not in what is stored.
- Both the dashboard chat and the widget. They run the same engine, so bunnyquery.com and the widget on your own pages search the same way.
Two habits make the walk shorter still. Keep a name your Bunny can search by at the start or the end of a file or table name, a client, a month, a product, since that is what "begins with" and "ends with" catch. And keep the column that names what you will ask about, an artist, a vendor, an account, as a real column in your sheets: the key values of a row are what become its tags, and a tag is the shortest path to every row about one thing.
Faster spreadsheet indexing
While looking at where the time goes, we measured the other half of the job: indexing. Drop eleven monthly settlement workbooks of about 550 rows each on the Files page and, until now, they took about half an hour, three minutes a file, one file after another.
The reading was not the slow part. The server reads a whole workbook in a second or two. The time was in the writing: rows go into your database in groups of ten, and each group was taking about seven seconds, because the writer and your project's database can sit in different regions and every row made its trips between them one after another. Six groups were in flight at a time, so a 550-row workbook spent about two and a half minutes just writing.
Two changes, both on the server, nothing in the widget:
- More groups in flight. The spreadsheet writer now sends 24 groups at once instead of 6. In our measurements the write step went from 7.6 rows a second to 21.7, with no errors and no throttling, and each group still took the same time to land, so the database absorbs it cleanly.
- Files side by side. A file's first pass used to hold the queue for the whole file. It now does one round of writing and hands the rest to its own continuation, so the queue is free to admit the next file, and up to three files are written at the same time.
What to expect: a workbook that spent about three minutes being written should take about one, and a batch of eleven monthly statements that took half an hour should come down to a few minutes. Nothing changes in what is written. The same rows land with the same sheet, row and grain fields, the same count of expected against written runs for every sheet, and a file is still never marked done on a shortfall. Within one group of rows the database still writes in the order given, for every app that writes to Skapi, exactly as before.
More news coming up!
Subscribe to the newsletter from your account profile page so you do not miss anything, and past articles are always at bunnyquery.com/articles.
Enjoy your Bunny!