Last night Ben gave a presentation on Data visualisation to the crowd at Ignite Melbourne.

Update – Video of Flink Labs’ talk at Ignite Melbourne is now available on youtube.

The PDF of his slides are available at http://flinklabs.com/slides/ignitemelbourne2010.pdf. Below are the preparation notes from the talk.

Thank you to all the organisers and audience for the wonderful event.

Lies, damned lies and MySchool
A talk given at Ignite Melbourne 2010 by Ben Hosken

Good evening. My name is Ben Hosken and I’m a founder of Flink Labs, a data visualisation studio based here in Melbourne. Tonight I’m going to talk about data visualisation, what it is, why its starting to take off and some of the potential problems we as viewers and consumers of data visualisations need to be aware of.

Data visualisation is the process of taking complex multivariate datasets and creating a visual representation of the data. Be it an interactive map, an animated infographic, or a tool for data exploration, data visualisation is at the sweet spot of computing, maths and design.

Data visualisation can be used as a tool for discovery and self exploration of datasets, potentially acting as the interface to the data. And it can also be used to communicate and engage your audience around a set of data.

All data has a point of view and a story to tell, often many stories. Data visualisation can be seen as the process of discovering and telling those stories. The stories can often be very good but even great story tellers can create bad stories.

Data visualisation can also be used as propaganda. Through deliberate misrepresentation and distortion, data visualisation can be used like many media as a tool to influence inappropriately.

So what’s happened recently that is causing the growth in data visualisation?

Our brains are hard wired to handle visual inputs. A very large amount of our brain, the visual cortex, is devoted to processing and understanding visual inputs. And our memory handles visual information better as well. We can recall information presented visually over 6 times as well as non visual information.

As growth in size and complexity of datasets in social networks, government, non-government and corporations accelerates, existing tools such as spreadsheets are no longer catering to the visualisation demands to the creators and audience.

The massive growth in computer power on devices, such as the iphone, upcoming ipad and the laptops we use on our desks, opens up the opportunity for rich interactive visualisations to be used to explore and engage with our data.

And people’s expectations are changing, as data rich experiences such as World of Warcraft improve people’s ability to process complex sets of visual data, the expectation is that these consumer experiences migrate into our day to day work and life.

The best data visualisations provide a point of view, guiding the audience around the data AND give the viewers the ability to interact with the visualisation crafting their own narrative for self exploration of the data.

But there can also be problems. Problems with the data, the way it is analysed, the way it is interpreted and the way it is represented. These problems can be created by mistake or can be intentional distortions aimed to misinform the viewer.

Over emphasis on one measure within the data, often “the average”, can lead to visualisations that ignore interesting outliers and emerging trends. It is important to understand the goal of the visualisation and the unique characteristics of the data when choosing what to visualise.

Appearing as an authoritative measure on a mainstream news show, this pie chart shown on Fox news in America is the worst example of data misrepresentation, not only is a 3D pie chart a poor method to use but in this case the raw data just doesn’t add up.

The question of “compared to what” is at the heart of any good visualisation. In this case, Senator Fielding has shown only the last 10 years of climate change data to bolster his point of view. By quoting data out of context, he distorts and misrepresents in this piece of propaganda.

Launched just over a month ago, the mySchool website gives parents of school aged children a way of viewing their schools NAPLAN test performance against nearby schools and schools of similar socio economic standing.

Based on a single 2 hour test taken every two years, the NAPLAN score provides an exceptionally narrow representation of what constitutes a good school.

With no ability to compare local schools, explore how different types of schools perform, see improvement over time and an overemphasis on averages, it has unfortunately created a service with lots of data but poor representation and data visualisation.

Creators have the responsibility to be ethical and not distort the data they are visualising. Numbers should add up, data should be quoted IN context, and the raw data should be meaningful and accurate.

And consumers of data visualisations should think critically about these new representations. Just as we have learnt to be critical of written pieces, we need to develop the skills to think visually and be critical of what can be a very powerful and persuasive medium.

In a time when we all seek to engage with our customers in more meaningful ways; data visualisation can help audiences to understand our products, services, and data in ways not previously possible.

As the volume and complexity of data increases, data visualisation can help you move beyond the endless rows and columns of faceless data, helping you communicate and engage with your audience. It’s time to move beyond the bar chart.

Thank You.

Last week marked the start of a new phase in the life of Flink Labs.

If I look back on the time since we started in May 2009, there have been 2 distinct phases, the first being from May until Oct during which time we got things set up, talked to potential clients, worked on our portfolio, got some great coverage, especially for the Ebb and Flow of Melbourne trains.

The second phase started in October when we presented Concept Lens, a Twitter visualisation, at Web Directions South. In some ways this was our coming out event. We had a small stand and a massive screen set up in the conference foyer and everyone had to walk past us to get to the talks. We got lots of leads out of the conference and from mid Oct until Christmas we have been engaged with a range of fantastic clients working on data visualisations, data mining and visualisation, conceptual user interfaces for visualisations and a couple of pure development projects. It was a real up-tick in the studio for the last quarter.

And now the third phase is beginning. After a good break, we have been back in the studio working with some amazing new clients with at least five solid projects in the pipeline, which if all came off would keep us exceptionally busy until the middle of the year. So this period is very much about execution and ensuring we can handle the traction we are getting in the marketplace.

In between and alongside these projects we are working on a few special interest pieces that are of relevance to us as a studio and also showcase some of the capabilities of data visualisation to a more mainstream audience.

One of these projects is our take on how data visualisation could be used in the recently released MySchool website that was launched by federal government a couple of weeks ago. Basically it is a reporting site for the all schools standardised testing results but unfortunately the site lacks a lot in terms of visualisations, comparative analysis and interactive self directed exploration. It’s very much a search and peck style service and we believe that it could be much much more. To that end we are working on visualisation of the data that we should have available in the coming 1-2 weeks.

We’re all very excited about the launch of the iPad and we’ve been thinking a lot about platforms/frameworks in which we want to focus moving forward. While we will continue to do work in Processing, we will be shifting our focus to the fantastic Protovis framework and starting to produce more work that runs natively in the browser be that on the desktop or on a mobile device.

So far, February has been fantastic and it looks like we’re going to have a great 2010 as our name gets known more in Australia and as our client list fills with fantastic companies.

A few days ago the UK’s Met office released a subset of global temperature readings from the last 200+ years.

Following on from Manuel Lima’s blog post, asking for the data viz community to get behind the climate change cause and use our skills to create some compelling visualisations, Jer Thorp has created a site at http://climatedata.blprnt.com/ for the data, where people have already contributed cleaned up data that can be directly loaded into MySQL along with some great code samples.

We’ve spent this afternoon exploring the data and creating a quick data viz showing the change in temperature readings over the last 200+ years. It’s not perfect and, personally, I believe that a well put together line chart may be the most compelling method for visualising this data.

A video and the source code, in processing, are available at http://flinklabs.com/projects/climatedata/.

We encourage you all to jump into this and hopefully the ease of access to the data along with good code samples gets everyone going.

Wow..it’s been a while since we updated our blog and plenty has been happening.

The last few months have seen a massive uptick in the amount of work we have been doing and the trend is, fantastically, looking like continuing well into the new year.

In October we attended Web Directions and showcased our twitter visualisation, Concept Lens, during the conference.The feedback was great, and importantly, we got lots of quality leads from people interested in engaging us to help with data visualisation of their data.

Almost immediately after the conference we were engaged with two pieces of work. The first for a very large energy company here in Australia to create a visualisation of their infrastructure assets around the country. Quite unlike the data intensive visualisations we have been familiar with this piece was a combination of mapping, images, movie making and voice overs resulting in a 7 minute data visualisation as movie that was presented to over 200 people internally.

The second was a more technical oriented piece of work with Telstra, assisting them, very briefly, in some development work required for the launch of their new set top box product due early next year.

In early November we submitted our entry to the Mashup Australia competition. Forgoing the omnipresent “Google Maps Mashup” we created a custom visualisation merging the weather in Melbourne in 2008 with air quality data during the same period from the EPA. Aside from some exceptionally obvious vote rigging, the competition has been fantastic for the development community in Australia with Hackdays held in Sydney, Canberra, and Melbourne.

The rest of November has seen us Toiling in the Data Mines, to quote Tom Armitage at Berg. We’ve been working with a new, and hopefully long term, client to undertake some hardcore data mining, clustering and sentiment analysis. This work has a huge amount of potential and this morning we delivered the initial results to the client. We’re exceptionally proud of what we have done in this piece of work, especially given the tight timeframes.

This week we get a chance for a brief pause before diving back into it with client presentations next week and initial meetings with some potential clients we would be very excited to work with.

Then, hopefully, it’s into implementation for a couple of projects over the Dec/Jan period leading into Feb and, probably, a trip to New Zealand for some conference connection making.

Today we were invited to present at Sensis‘ Future Friday workshop, talking about Data Visualisation and Search.

Thanks to everyone who attended and presented, it was a great way to end a very busy and productive week.

Our slides are here.

Just over two weeks ago we launched our latest piece of work, Concept Lens, at the most excellent Web Directions conference in Sydney, Australia.

Web Directions is Australia’s leading web focused conference and Maxine, John and their team make a fantastic effort at doing their bit to help develop the web industry here in Australia.

We approached Maxine back in August about working together to produce a visualisation of the “back channel” conversation that was occurring during the conference. In today’s technology that means Twitter for the conversation and Flickr for the photos. I remember back to 2004 and Joi Ito presenting the backchannel on IRC for eTech on a scrolling LED display on stage. Times have changed and the richness of conversations, topics and visualisations have developed.

Over the month prior to the conference and between client work, we pulled together our “thumb in the pie” Twitter visualisation. We had seen many abstract pieces visualising Twitter and while we wanted to make sure we delivered something aesthetically pleasing, it was equally important that the application and associated visualisation was useful for the audience to gain insight into the back channel conversations around the event.

Using our tool of choice, Processing, we proceeded to sculpt and sketch some initial ideas of how the visualisation could work, and along the same lines as the recent excellent article on working in the data mines by Berg, we always find that until we play, explore and get our hands dirty with the datasets we can’t come up with the appropriate design or visualisation. Concepts that we think would work often don’t, schemas in the data at first glance look appropriate and complete but turn out to be null or incomplete.

Our initial data explorations focused on the Twitter search feed and understanding what was in the data and how we could use it. One of the concepts I have been very keen on since first hearing about it from Jeff Jonas in 2007 is that of “data finds data”. Jeff has written plenty about this topic but here is a brief example, “you have two records that refer to the same person, but you don’t know that they do. Then a third record appears which relates to each of the first two, and which establishes that all three refer to the same person. The first two pieces of data find one another, through the agency of a third piece of data.”

After developing an understanding of the Twitter search result data and the inconsistencies between schema and actual results, along with getting a “feel” for the data inside the results, we proceeded to sketch out some initial visualisations. A key concept was being able to show data over time and in this case, the representation we were drawn to was the use of “zoomable” timeline as away of pegging the data in time and space. The zoom interface gives the user a macroscope-style tool which allows them to see activity over a 10 minute period up to a three day window giving a quick visual cue regarding hot points and areas of great discussion.

Once we had the basic rendering happening, we shifted to the key focus for the project, that of identifying relationships between the snippets of conversation. Back to the data finds data concept here. While the volume of tweets was by no means huge, we decided to develop and approach that didn’t require the application to revisit every piece of data each time a new tweet came in just to find relationships that may or (most likely) may not exist.

To this end, we created some structures that supported key lookups driven off plugins that understand a specific concept around the tweets, such as a RT, a conversation thread, a hashtag topic, a keyword, or an author, and let them handle the relationship building as each new tweet comes into the application.

This is turned out worked exceptionally well, both fast enough to actually be done on the fly and customisable enough that we could, and did, implement new plugins with 30 mins work.

The result of this was a zoomable time line based visualisation of the conversation around the event that enabled the viewer to navigate the large set of tweets and view the relationships between them in an efficient and useful manner.

It’s by no means perfect, but we did achieve the two goals of delivering something that was well accepted at the conference and creating a tool that we continue to find ourselves using to gain insight into specific events such as sporting matches and conferences.

Of course it wasn’t all sunshine and roses, we experienced a lot of pain dealing with the native OpenGL calls required to get any performance out of Processing, especially around text display and, unexpectedly, the rendering of “points” for each tweet. Inconsistent OpenGL problems took up a few days of development time but we did end up learning a heap about monitoring the performance of Processing optimising it using appropriate OpenGL calls. The lack of GUI widgets in Processing also meant we had to roll our own components for everything on the screen and while interesting intellectual exercise it was slightly painful in the last few days as we crunched to deliver.

We learnt heaps from actually seeing the visualisation in use in the wild and getting some wonderfully constructive feedback from many of the conference attendees.

We’ve updated the Concept Lens code to enable it to be used around any topic or search term. It’s available for Mac and PC at www.flinklabs.com/projects/conceptlens

We received a LOT of interest in using Concept Lens to monitor conversations around brands and several conference organisers are very keen to use it in upcoming events. So the next steps are to investigate integrating some sentiment analysis into the visualisation, probably moving it to Flex and to polish up some of the rough edges taking into consideration the learnings we got from Web Directions. We’ll probably also remove Flickr as it, somewhat expectedly, gets used after the event as people return home, sort out their photos and uploads them removing the immediacy of the images.

Finally, we really want to thank the team at Web Directions for supporting us in implementing Concept Lens and giving us the space and facilities at the conference. Also, Jeremy Yuille was exceptionally helpful during the ideation phase and provided many use focused insights that helped clarify a lot of our thinking.

This week we had the pleasure to be invited to a workshop about developing data visualisations to support the Australian national curriculum in schools held by the Learning Federation.

It was an exceptionally interesting group with plenty of GIS representatives, exceptionally experienced teachers and a couple of data visualisation people…we were the minority to be sure.

Everyone was exceptionally open and I’m sure the organisers got a huge amount out of it.

For those interested, here are my slides.

We’ve been quite busy in the studio over the last few weeks. We’ve got some deadlines coming up in October and need ti stay focused to ensure we meet them.

What we were up to at Flink Labs in the last week or so.

• Building initial versions of our Twitter/Flickr visualisation
• Finishing our government 2.0 submission…more on this soon
• Chilling out to Morcheeba
• More astronomy at the Sydney observatory
• Ordering some fun publicity stuff for new projects

Business Week magazine features an article by Maria Popova giving a brief overview and examples of some recent data visualisations.

The article tends to focus on the more aesthetically oriented pieces but is a good overview of attractive looking pieces and smaller set of useful visualisations such as crime spotting.

She ends the article with some brush stroke examples of where data visualisations could be used for within a business context. While by no means a full coverage it does spark some potential opportunities, especially taking large sets of data that are shown to consumers as tables and using a visual representation to improve that communication be it energy, telecom or banking.

At Flink Labs we're very bullish on using data visualisation to engage with and communicate with your customers and stakeholders. Aesthetically attractive pieces such as Melbourne Trains expose a complexity hidden from the average commuter while the H1N1 visualisation engages with the population to, hopefully, put the number of deaths from swine flu in context.