There are three things that could put business intelligence capabilities at everyone's fingertips, but will they ever happen? A long-suffering industry observer offers his wish list for BI software.
I was reading a business intelligence (BI) white paper feed when I happened across one whose title was, more or less, "Data Visualization: Is This the Way To Attract the Common End User?" And I thought, "Oh boy, here we go again."
You see, the idea that just a little better user interface will finally get Dick and Jane employee to use databases dates back at least 26 years. I know, because I had an argument with my boss at CCA, Dan Ries (a very smart fellow), about it. He was sure that with a fill-out-the-form approach, any line employee could do his or her own ad-hoc queries and reporting. Based on my own experiences as a naïve end user, I felt we were very far from being able to give the average worker an interface that he or she would be able or motivated to use. Here we are, 26-plus years later, and all through those years, someone would occasionally pipe up and say, in the immortal words of Bullwinkle, "This time for sure!" And every time, it hasn't happened.
I divide the blame for this failure equally between vendors' marketing organizations and IT buyers. Database and BI vendors, first and foremost, look to extend the ability of specific targets within the business to gain insights. That requires ever more sophisticated statistical and relationship-identifying tools. The vendor looking to design a "common person" user interface retrofits the interface to these tools. In other words, despite a public devotion to the common touch, the vendor acts like it is selling to a business expert, not a consumer, market.
Meanwhile, IT buyers looking to justify the expense of BI try to extend its use to upper-level executives and business processes, rather than demand that vendors extend the interface approach of popular consumer apps to using data, or that it give the line supervisor who uses it at home a leg up at work. And yet that is precisely how Word, Excel, maybe PowerPoint, and Google search wound up being far more frequently used and useful than SQL or OLAP.
I have been saying things like this for the last 26 years, and somehow the problem never gets solved. At this point, I am convinced that no one is really listening. So for my own amusement, I give you three ideas — proven in the real world, but never implemented in a vendor product — that if I were a user I would really like and that I think would come as close as anything can to achieving "BI for the masses."
Wayne Kernochan of Infostructure Associates has been an IT industry analyst focused on infrastructure software for more than 20 years. Article used by permission of Focus.
<h1>BI Idea 1: Google Exploratory Data Analysis</h1>
I'll start with a long example of what I'm looking for here:<p>
I'm reading through someone's blog when they mention "graphical analysis." What the hey? There's a pointer to another blog, where they make a lot of unproven assertions about graphical analysis. Time to visit Google: a search on graphical analysis results in a lot of extraneous information, some of it very interesting, plus Wikipedia and a vendor who is selling this stuff. Wikipedia is off-topic, but carefully reading the article shows that there are a couple of other articles that might be on point. One of them gives me some of the social-networking theory behind graphical analysis, but not the products or the market.<p>
Back to Google, forward to a couple of analyst market figures. They sound iffy, so I go to a vendor site and get their financials to cross-check. Not much in there, but enough that I can guesstimate. Back to Google, change the search to "graphical BI." Bingo, another vendor with much more market information and ways to cross-check the first vendor's claims. Which products have been left out? An analyst report lists the two vendors, but in a different market, and also lists their competitors. Let's take a sample competitor: what's their response to "graphical analysis" or graphical BI? Nothing, but they seem to feel that statistical analysis is their best competitive weapon. Does statistical analysis cover graphical analysis?<p>
The names SAS and SPSS keep coming up in my Google searches, but it doesn't seem as if their user manuals even mention the word "graph." What are the potential use cases? Computation of shortest path? Well, only if you're driving somewhere. Still, if it's made easy for me... Is this really easier than Mapquest? Let's try a multi-step trip. Oog. It definitely could be easier than Mapquest. Can I try out this product? All right, I've got the free trial version loaded, let's try the multi-step trip. You know, this could do better for a sales trip than my company's path optimization stuff, because I can tweak it for my personal needs. Combine with Google Maps, stir ... wouldn't it be nice if there was a Wikimaps, so that people could warn us about all these little construction obstructions and missing signs? Anyway, I've just given myself an extra half-hour on the trip to spend on one more call, without having to clear it.<p>
Two points about all this. First, Google is superb at free-association exploratory analysis of documents. You search for something, you alter the search because of facts you've found, you use the results to find other useful facts about it, you change the topic of the search to cross-check, you dig down into specific examples to verify, you even go completely off-topic and then come back. The result is far richer, far more useful to the "common end user" and his or her organization, and far more fun than just doing a query on graphical data in the company <a href=http://www.webopedia.com/TERM/D/data_warehouse.html>data warehouse</a>.<p>
Second, Google is lousy at exploratory <em>data</em> analysis, because it is "data dumb" — it can find metadata and individual pieces of data, but it can't detect patterns in the data, so you have to do it yourself. If you are searching for "graphical analysis" across vendor websites, Google can't figure out that it would be nice to know that 9 of 10 vendors in the market don't mention "graph" on their websites, or that no vendors offer free trial downloads.<p>
The answer to this seems straightforward enough: add "guess-type" data analysis capabilities to Google. And, by the way, if you're at work, make the first port of call your company's data warehouse data store, full of data you can't get anywhere else. You're looking for the low-priced product for graphical analysis? Hmm, your company offers three types through a deal with the vendor, but none is the low-cost one. I wonder what effect that has had on sales? Your company did a recent price cut; sure enough, it hasn't had a big effect. Except in China; does that have to do with recent exchange rate manipulations, and the fact that you sell via a Chinese firm instead of on your own? It might indeed, since Google tells you the manipulations started three weeks ago, just when the price cut happened.<p>
Get the idea? Note that the search/analysis engine guessed that you wanted your company's data called out, and that you wanted sales broken down by geography and in a monthly time series. Moreover, this is <em>exploratory</em> data analysis, which means that you get to see both the summary report/statistics and individual pieces of raw data to see if your theories about what's going on make sense.<p>
In Google exploratory data analysis, the search engine and your own exploration drive the data analysis, not the tools available and supposedly designed for this process. It's a fundamental mind shift, and one that explains why Excel became popular and in-house on-demand reporting schemes didn't, or why Google search was accepted and SQL wasn't. One is about the features; the other is about the consumer's needs.<p>
Oh, by the way, once this takes off, you can start using information about user searches to drive adding <i>really</i> useful data to the data warehouse.<p>
BI Idea 2: The Do the Right Thing Key
Back in 1986, I loved the idea behind the Spike Lee movie so much that I designed an email system around it. Here's how it works:
You know how when you are doing a "replace all" in Word, you have to specify an exact character string, and then Word mindlessly replaces all occurrences, even if some should be capitalized and some not, or even if you just want whole words to be changed and not character strings within words? Well, think about it. If you type a whole word, 90 percent of the time you want only words to be replaced, and capitals to be added at the start of sentences. If you type a string that is only part of a word, 90 percent of the time you want all occurrences of that string replaced, and capitals when and only when that string occurs at the start of a sentence. So take that Word "replace" window, and add a Do the Right Thing key (really, a point and click option) at the end. If it's not right, the user can just Undo and take the long route.
The Do the Right Thing key is a macro; but it's a smart macro. You don't need to create it, and it makes some reasonable guesses about what you want to do, rather than you having to specify what it should do exactly. I found when I designed my email system, every menu, and every submenu or screen, would benefit from having a Do the Right Thing key. It's that powerful an idea.
How does that apply to BI? Suppose you are trying to track down a sudden drop in sales one week in North America. You could dive down, layer by layer, until you found that stores in Manitoba all saw a big drop that week. Or, you could press the Break in the Pattern key, which would round up all breaks in patterns of sales, and dig down not only to Manitoba but also to big offsetting changes in sales in Vancouver and Toronto, with appropriate highlighting. Nine times out of ten, that will be the right information, and the one other time, you'll find out some other information that may prove to be just as valuable. Now do the same type of thing for every querying or reporting screen.
The idea behind the Do the Right Thing key is actually very similar to that behind Google Exploratory Data Analysis. In both cases, you are really considering what the end user would probably want to do first, and only then finding a BI tool that will do that. The Do the Right Thing key is a bit more buttoned-up: you're probably carrying out a task that the business wants you to do. Still, it's way better than "do it this way or else."
BI Idea 3: Build Your Own Data Store
Back in the days before Microsoft Access, there was a funny little database company called FileMaker. It had the odd idea that people who wanted to create their own contact lists, their own lists of the stocks they owned and their values, their own grades or assets and expenses, should be able to do so in whatever format they wanted. As Oracle steadily cut away at other competitors in the top end of the database market, FileMaker kept gaining individual customers who would bring its application into their local offices and use it for little projects. To this day, it is still pretty much unique in its ability to let users quickly whip up small-sized, custom data stores to drive, for example, class registrations at a college.
To my mind, FileMaker never quite took the idea far enough. You see, FileMaker was competing against folks like Borland in the days when the cutting edge was allowing two-way links between, let's say, students and teachers (a student has multiple teachers, and teachers have multiple students). But what people really want, often, is "serial hierarchy." You start out with a list of all your teachers; the student is the top level, the teachers and class location/time/topic the next level. But you next want to see if there's an alternate class; now the topic is the top level, the time at the next level, the students (you, and if the class is full) at a third level. If the number of data items is too small to require aggregation, statistics, etc., you can eyeball the raw data to get your answers. And you don't need to learn a new application like Outlook, Microsoft Money or Excel for each new personal database need.
The reason this fits BI is that, often, the next step after getting your personal answers is to merge them with company data. You've figured out your budget, now do "What if?" scenarios: Does this fit with the company budget? You've identified your own sales targets, so how do these match up against those supplied by the company? You download company data into your own personal workspace and use your own simple analysis tools to see how your plans mesh with the company's. You only get as complex a user interface as you need.
Will Any of This Ever Happen?
I hope you enjoyed these ideas, because, dollars to doughnuts, they'll never happen. It's been 25 years, and the crippled desktop/folder metaphor and its slightly less crippled cousin, the document/link browser methodology, still dominate user interfaces. It's been fifteen years, and only now is Composite Software's Robert Eve getting marketing traction by pointing out that trying to put all of a company's information in a data warehouse is a fool's errand. It's been almost 35 years, and still no one seems to have noticed that seeing a full page of a document you are composing on a screen makes your writing better. At least, after 20 years, Google Gmail finally showed that it was a good idea to group a message and its replies. What a revelation!
No, what users should really be wary of are vendors who say they do indeed deliver any of the ideas listed above as actual, workable solutions. They don't; but saying they do is a bit like other vendors' claims that requirements management software is an agile development tool. No; it's a retrofitted, slightly less sclerotic tool instead of something designed from the ground up to serve the developer rather than the process.
But if you dig down deep, and the vendor really does walk the walk, grab the BI tool. And then let me know the millennium has finally arrived. Preferably not after another 26 years.