Tuesday, February 7, 2012

Wow! More Than 1 Million Rows of Data per Minute!

As we continue to work on Dodeca, we continue to stretch the limit.  Today was certainly evidence of that. Dodeca already has the most powerful relational/spreadsheet technology available in the world, but our development team is not yet satisfied.  Instead, they are working on making it even better.  Specifically, they have implemented a cursored retrieval that will allow the Dodeca client to request a specified number of rows on each trip to the server.

To test this functionality, our dev team created some very large retrieves.  The SQL itself was fairly simple in that it returned only 7 columns of data and joined only 2 tables.  The performance we saw, however, was outstanding.  In tests conducted on a debug build of Dodeca, they saw over 1,030,000 rows of data, or a total of over 7,200,000 cells of data, retrieved from SQL Server and filled into a worksheet in about 37 seconds.  That means Dodeca is processing nearly 195,000 cells per second!

Many people ask me, "Why would you want to retrieve 1,000,000 rows of data into a spreadsheet?"   Well, we have customers who retrieve massive amounts of data.  We believe that, as a software provider, it isn't our place to second-guess why a customer wants to do something like that.  Instead, we focus on allowing customers the freedom to do their business their way; Dodeca just makes it faster and easier.  At 195,000 cells per second, I would call it really fast!  How would you like to see performance like that when you do an Essbase drill-through?


srx said...

Hi Tim just a question, is this faster than classic Add-in? Well it looks like, would it be because Dodeca is installed close to Essbase engine and then benefits from direct access to APIs and application server?

Tim Tow said...

Hi Sébastien,

In terms of raw cells/second, the classic Add-in is still a little faster but it really isn't an apple-to-apple comparison. Dodeca also returns attribute information about each cell in addition to layout information. For example, in Dodeca, the row header, column header, page header and data ranges are automatically identified for you in a range name. Further, information about the layout and the position of each dimension is also returned. Finally, Dodeca uses SOAP/XML for transport over HTTP, so it can retrieve data over the open internet (via HTTPS) without a direct network connection to the Essbase server.

BTW, we enjoyed having your friends from Limpida working here for a week. The discussions gave us some further ideas on how we can make Dodeca even better!