Whats new with Big Data?

Big data… We hear about it literally everywhere these days but what is it, really??

This past spring I was contacted by a specialist network I am engaged in (Informed Decisions). They asked if I, on very short notice, could participate as a speaker at a seminar arranged by Embarcadero (http://www.embarcadero.com/)  in Manchester. The brief was short and swift; “just talk about your ‘standard’ information modeling and taxonomy management approaches”. “OK, I can do that. 40 minutes, you said? Right, no problem!”

Next call was from the seminar planner at Embarcadero. Disregarding the 99% super nice chatting during the call, the comment “Bear in mind that the audience is expecting talks to address Big Data and Information Architecture content” almost made me choke. I called back to ID… “Listen guys, you know I am not a Big Data Architect…”

Let’s spare you the details of the calls that followed, but somehow they all managed to convince me that it all was a really good idea. Personally I felt like I was about to enter the lions’ pit in ancient Collosseum. Not only do I not know that much about database models, I also tend to resent databases as I work mainly with unstructured information and find the theories behind database architectures too limiting. In either case, this truly forced me to start thinking about what “Big Data” really is and what it means to us.     

After a rather quick read through of a number of papers on the matter I came to an “astonishing conclusion”. My revelation: Big Data simply equals an obscene amount of data… Joke aside, it is quite clear that the label itself, rather than referring to the actual numbers of Petabytes, is a label set on the challenges we face in dealing with these obscene amounts of data, very much from a technology and hardware perspective. 

Reading on, I also came across some analyst reports on the topic that added to the same conclusion but also brought in a perspective which, to me, is more interesting, namely that of how to use the Big Data. There was one particular quote that really caught my attention: “All the data in the world doesn’t help if the right questions aren’t asked, and big data does not generate such questions, or even contribute to their formulation.” (Jones/Silberzahn, Forbes Leadership 7 feb 2013). It reminded me of another, more than a decade old, quote (source forgotten) saying roughly that “the volume of information which an individual in an eighteenth century village encountered during a lifetime about the world outside the village equals one issue of the Financial Times.” This, of course, is an allegory of sorts but with the purpose of putting us humans in the center of the information tsunami we are, to a greater and greater extent, experiencing.

The main theme of my presentation hence ended up being that when it comes to humans, Big Data is nothing new whatsoever. The challenge has been with us for decades and to the human brain, the difference between far too much and obscenely too much is nil. We just cannot handle it. Hence, from a human usage perspective, we can still approach the data volumes with the same models and tools we have used for some time now. The true super power of Big Data, however, comes when we can start combining such practices with the new models growing out of the sheer hardware and software needs driven by the obscene amounts of Petabytes that need to be handled. I further realized, during my “research” prior to the presentation, that if this is to happen, people like me and true BigData/architecture people need to get together and start bouncing ideas off each other. Maybe that is already happening everywhere, but for me this was a first encounter. I have continued my “quest for more understanding in the field” by doing just that and it truly seems like there are lots of learnings to be gained in both directions so let’s start  bouncing ideas off each other!

So, did the presentation go down well? This is not for me to judge but from the feedback I got, yes it did.

As always, I’d love to hear your thoughts about this below.