Through the wormhole with Morgan Freeman

by seangourley on September 18, 2013

One of the fun things about doing the kind of research that I do is that sometimes Morgan Freeman** shows up with a Television crew in tow. Earlier this year I was lucky enough to be able to put together a show with him about some of the more cutting edge ideas of free will and determinism. The idea being that although we may have some freedom when it comes to an individual within society, once you scale these kinds of systems up towards millions the aggregate behavior starts to become quite predictable. While freedom may exist for an individual, do we indeed lose it as a society?

This question is particularly interesting when we start to consider the mathematical structures that appear time and again within insurgencies and war. From the timing of attacks through to the size distribution of attacks. The activities start to become ordered and predictable. When this order emerges do we lose some sense of free will? If even the way we choose to kill each other within a seemingly chaotic conflict is determined by a set of mathematical equations what does that say about our ability to collectively have free will?

Of course this is all nothing new to the Issac Assimov fans out there, and the fictional Professor Hari Seldon was/is a large proponent of galactic scale predictability with his version of psychohistory. With the emergence of global scale data collection and high performance computing systems available on demand – are we finally starting to brush up against this strange/scary concept?

I’m not sure I did quite enough Philosophy to answer  this or any of the other challenging questions definitively. But it was certainly an interesting set of questions to ponder. It was a blast working with the ‘Through the Wormhole’ team, and great to be able to tell the story of the work I have been doing into understanding the mathematical structure of war and insurgency. And judging by the video you can see that is was also a great excuse to lace up the running spikes, get onto the track and get out over some hurdles — judging by the technique it’s been a while since I had this chance. You can watch some of the other clips from the show here including an interesting piece from Michael Gazzaniga aka the father of modern neuroscience.

**And because I know you are asking, I did of course get Morgan Freeman to record for me a voice message on my answer phone. Is there any other reason to work with him?

{ Comments on this entry are closed }

art of propaganda
Late last year I was sitting in Osaka in a little bar in Dotonbori, watching the presidential election results come through. The television was tuned to the Japanese national broadcaster NHK and from time to time different results would come through. Obama kept piling up wins across the country until at around 1:30pm Japanese time CNN called the election for Obama. This result, like most of the earlier Electoral College results, was met with a muted response. A mild round of polite applause and then back to eating, drinking and text messaging friends. The relative indifference to the result couldn’t have been further from the hype my friends were experiencing first hand as they watched in New York and San Francisco. But as I watched the results my emotional levels mirrored that of the Japanese salaryman next to me in the bar – I felt very little in the way of anticipation or surprise. I had followed Nate Silver’s models over the past 6 months and as the Election Day drew closer and closer the models became more and more accurate. Checking in to the website fivethirtyeight.com on the Friday before the election and it was clear from the data and models that Silver had built that Obama would have an estimated 86% chance of winning. As the results came through, the models proved to be correct — there was simply nothing to be surprised about. It was the political equivalent of being up by 6 runs at the top of the 9th at the Giants baseball game — time to go home and beat the traffic.

The truth is that we are getting very good at predicting things like elections and the closer we get to the election day, the more accurate the prediction we can make. The models that power these predictions have been becoming increasingly more complex over the past decade and for someone like Nate Silver this improvement process has involved thousands of hours of work to get to what could be considered a v3.0 of his election model. Add to this an exponential growth in available data and on demand distributed cloud computing engines and the predictive power of these models start to become impressive. Three days out from the presidential election in the United States we can predict the likelihood* of either candidate winning — we have the technology. Get over it. Predicting election results 72 hours out will be as routine** as predicting the weather — and just as with predicting the weather, we will get much better with it over time.  It’s fair to say that with electoral prediction we are only at the beginning of this upward facing curve.

For those of us in the data science field, the ability to predict election results comes as no surprise. As a group we spend our days trying to predict everything from which ads a Facebook user will click on, to which way the stock market will move in the next 1/10th of a second to the perfect movie to watch on a Wednesday evening. Which is to say nothing of our collective efforts to predict the evolution of conflicts, the movement of people on a daily basis and the outbreak of new viruses. Over the last 10 years we have got very good at predicting human behavior. Over the next 10 years the rest of the world will realize the implications of this — and a lack of election surprises is one of them.

[click to continue…]

{ Comments on this entry are closed }

Radio New Zealand Interview

by seangourley on May 21, 2013

New Zealand-born physicist Sean Gourley on the mathematical patterns that underly war, global intelligence, and his roles as a political advisor to the Iraqi government, the UN and the Pentagon.

{ Comments on this entry are closed }

Every year the Cartography and Geographic Information Society holds a competition – the equivalent of the Academy Awards for maps — for the best map of the United States. While it’s often won by one of the major players in the mapping world, like the US Census bureau, in 2010 it was won by a one-man shop run by David Imus of Eugene Oregon.

Imus’s map differed not just in the scale of operation, but in the very way he went about constructing it. Traditional map-makers make use of algorithms to position labels, size towns and arrange points of interest, and they farm out the rest of the work to teams in India to manually fill in. While Imus’ map was constructed on a computer it didn’t use algorithms, leading to Imus toiling 6,000 hours, 7 days a week, for two years, obsessing over font types, state boundary colors and things like what symbol to use for airports. The little touches made the difference – the map was beautiful.

Imus, his map and his technique for producing it, seem like an anachronistic throw back to a 1950s world of exacto knives. But his approach is both an example of the future of information and a reminder of its past. As the world is under going an explosion of big data, it’s the multi-dimensional map interface that will play the key role for displaying connected intelligence. As Imus’ showed, producing these maps will require art and algorithms to come together.

Finding intelligence in Big Data

Much has been written about the proliferation of data over the past few years. Big data has been compared to the new oil, and like with oil, the environmental footprint of data is being felt with data warehouses now consuming around 2 percent of the electricity in the US.

But data in and of itself is useless — just a pile of 1′s and 0′s stacked together in server farms dotted across the nation. For data to actually be useful it needs to have algorithms run on top of it, and these algorithms need to lead to decisions. These decisions could be small — like which ad should I insert at the start of a YouTube video — or large such as should I insert 30,000 new troops into Iraq.

[click to continue…]

{ Comments on this entry are closed }

Breaking apart drug cartels?

by seangourley on October 25, 2012

“Killing drug lords gets headlines, but complexity analysis suggests they are the wrong people to target to bring down a cartel”.

New Scientist put together an interesting piece called ‘Destroying drug cartels, the mathematical way’. The article focussed on the recent death of Lazcano the leader of the Mexican cartel Los Zetas and on how mathematics, modeling and complexity theory are being used to fight drug wars like these.

Mexican drug ecosystem

My research looked at similar drug fueled conflicts and we found that the mathematical signatures that defined the violence in places like Colombia looked very similar to the signatures in more traditional type insurgencies like Iraq. The way that drug cartels organise themselves, evolve and compete with each other is very similar to the dynamics seen within insurgent groups in places like Iraq and Afghanistan. This is because there are only a few effective ways to organise a force against a conventional army/paramilitary unit – and either the groups evolve to find this solution or they die trying. The upside of this though is that strategies employed in Iraq to break apart insurgencies can be used to inflict damage to the Mexican drug ecosystem.

[click to continue…]

{ Comments on this entry are closed }

A few months back I got an email from Bruce Cahan who was putting together a TEDx event called TEDxNewWallStreet. The purpose of the event was to challenge some of the fundamental paradigms of Wall Street, and Bruce wanted to know if I would put together a lecture for it based on my work with algorithms and Big Data. The only requirement is that the content was somewhat experimental, with bonus points if it was a touch controversial. Experimental and controversial are two great motivators for me so I told him I was a ‘maybe’. I spent the next week bouncing around ideas with some of my favorite thinkers in Silicon Valley, reading journal articles, sketching notes, analyzing equations and drinking a lot of coffee down at cafe Trieste. At the end of the week I finally came up with something that I wanted to talk about. A talk at the intersection of High Frequency Trading, machine readable news, and predator-prey ecosystem stability. A talk challenging our reliance-of and our control-by algorithms.

That was the video, click through to read more of the story behind the ideas
[click to continue…]

{ Comments on this entry are closed }

A little story about Quid

October 5, 2011

While building Quid, in the beginning, one of the questions that was always hardest to answer was ‘what does your company do’. This was hard because there really isn’t a short soundbite answer to the question. How do you explain the collection of data, structuring of information and visualization of multi-dimensional space in 30 seconds? [...]

Read the full article →

New projects, new ideas

October 5, 2011

It’s been a little while since I last got the chance to sit down and write here. But for the last 18 months or so I’ve been immersed in a new project that is called Quid. Quid is the company I started back in Dec 2009, it is now at 60 people and we’re doing [...]

Read the full article →

Nature paper on the Russian cable channel RT

January 9, 2010

My colleague and coauthor Neil Johnson makes an appearance on the Russian cable channel RT. In the 6 minute video he talks about the results of the Nature paper and looks at how the model can be used to inform strategic decisions.

Read the full article →

14 key features that define a successful insurgency

December 21, 2009

Insurgencies are by their very nature difficult to understand. However each time an attack is launched and every time an IED explodes we start to know a little more about the structure of an insurgency. If we combine together enough of these attacks we start to build up a mosaic picture of the insurgency. Their [...]

Read the full article →