While building Quid, in the beginning, one of the questions that was always hardest to answer was ‘what does your company do’. This was hard because there really isn’t a short soundbite answer to the question. How do you explain the collection of data, structuring of information and visualization of multi-dimensional space in 30 seconds?

Over time though we got better at telling the Quid story, and over time the rest of the world has started to understand more and more of the technical elements of the Big Data space. The basic idea behind Quid is this: we are living in an increasingly complex world, with millions of interacting components and feedback loops, a world that may be beyond our raw human ability to understand it. With this in mind we set out to build a new set of tools to enhance the human ability to understand this complexity. Think of Quid as the new AI — not Artificial Intelligence, but instead Augmented Intelligence. I could go into more details, but I think this video does a pretty good job of telling the Quid story…..enjoy.

It’s been a little while since I last got the chance to sit down and write here. But for the last 18 months or so I’ve been immersed in a new project that is called Quid. Quid is the company I started back in Dec 2009, it is now at 60 people and we’re doing some pretty amazing things with data, mathematics and visualization. At Quid we are building a global intelligence platform, a place where open source intelligence is collected, structured and visualized to help people understand and make better decisions about the complex world we live in. It’s been an interesting transition out of the academic world and into the world of startups, products and venture capital. But it has been a move that has allowed me to scale up an awesome team, raise a significant amount of money and engage directly with real world problems. Now that Quid is coming up to it’s 2 year mark I hope that I will get more time to sit and write here and share some of my thoughts, ideas and theories that are driving what we are doing here at Quid. Check out Quid.com for more information about what we are doing.

My colleague and coauthor Neil Johnson makes an appearance on the Russian cable channel RT. In the 6 minute video he talks about the results of the Nature paper and looks at how the model can be used to inform strategic decisions.

Insurgencies are by their very nature difficult to understand. However each time an attack is launched and every time an IED explodes we start to know a little more about the structure of an insurgency. If we combine together enough of these attacks we start to build up a mosaic picture of the insurgency. Their actions can start to be defined mathematically and we can work backwards from these signatures to understand the fundamental forces that underlie the insurgency. This is exactly what we did in our latest research study “Common Ecology Quantifies Human Insurgency“.

With these models we can for the first time quantitatively understand more about what makes an insurgency successful. From our analysis and modeling we find that there are 14 key characteristics that define a successful insurgent ecosystem; these are listed below with a short name to describe the feature.

1. Many body: There are many more autonomous insurgent groups operating within conflicts than we had previously thought. For example there are 100+ autonomous groups operating in Iraq (as of 2006).
2. Fluidity: The insurgents are loosely grouped together to form fluid networks with short half-lives. This is very different from the rigid hierarchical networks that have been proposed for insurgent groups.
3. Redundancy: If we remove the strongest group from the system another group will rise to replace the previous strongest group
4. Splinter: When a group is broken it does not generally split in half but instead shatters into multiple pieces
5. Redistribute: When a group is broken the components are redistributed amongst the other groups in the system. The redistribution is biased towards the most successful remaining groups.
6. Snowball: The strongest groups grow fastest
7. Tall poppy: The strongest groups are the predominant targets for opposition forces
8. Internal competition: There is direct competition amongst insurgent groups for both resources and media exposure. They are competing with each other in addition to fighting the stronger counterinsurgent forces.
9. Independent co-ordination: Autonomous groups act in a coordinated fashion as a result of the competition that exists between them.
10. Emergent structure: Attacks in both Iraq and Colombia become ‘less random’ and more coordinated over time
11. Evolution: The strategies employed by the groups evolve over time where successful groups/strategies survive and unsuccessful strategies/groups are replaced.
12. High dimensional: Connection occurs over high dimensions (i.e. Internet, cell phone etc) and is not dominated by geographic connections.
13. Non-linear: It is approximately 316* times harder to kill 100 people in an attack than it is to kill 10 people. (*Results for a conflict with alpha=2.5).
14. Independent clones: the fundamental structure and dynamics of insurgent groups is largely independent of religious, political, ideological or geographic differences.

What can we learn from insurgents? Should the US military adopt more of these principles? Can we apply these organizational characteristics to other problems? You can read more about the research over at the TED blog, including the in depth interview I did with them.

I have been working over the last few days with my team to put together the new website mathematicsofwar.com. This website contains a set of resources to help people better understand the details of our research. It is more science/math focused site and includes copies of our latest research papers, background reading, working drafts and details of the modeling and analysis. We’ll be adding to this siteover the coming weeks and months and building it up as a resource for the quant conflict analysis space.

It’s quite a lot of literature to get through and covers a range of different topics including economics, statistics, physics and computer science. But then that’s what the holidays are for right :)

It looks like our research just got picked up on Slashdot. How long before quant analysis of conflict is also on the daily agenda of policy/strategy/military types.

To explain what was driving this common pattern, the researchers created a mathematical model which assumes that insurgent groups form and fragment when they sense danger, and strike in well-timed bursts to maximize their media exposure. Johnson is now working to predict how the insurgency in Afghanistan might respond to the influx of foreign troops recently announced by US President Barack Obama

Hello to any new visitors here via Slashdot.

A good post from John Robb over at Global Guerrillas about the “Ecology of war” Nature paper. Robb argues that the research put forward in the study has applications to what he calls Open Source Warfare. Robb believes that Open Source Warfare has become the dominant form of conflict for the 21st Century and it is defined as

a model of warfare that describes how many small autonomous groups can fight an insurgency despite the lack of a central command hierarchy.

This is an interesting theory, however up until now there has been little in the way of quantitative evidence to explain how insurgent forces could operate in such a decentralized fashion. Robb goes on to say that our model provides quantitative evidence for the mechanisms that would allow Open Source Warfare to successfully operate. These are;

A grouping mechanism. Why groups fragment and form. It assumes a constant insurgent population with a fluctuating number of groups depending on counter-insurgency pressure (more pressure = more groups).

A timing mechanism. A description of how insurgent decision making cycles (assuming lots of groups) impact the timing of attacks. The conclusion is that of cross group communication through the media (stigmergic learning) explains how this mechanism works.

These ground level mechanisms or rules of interaction provide a global structure to an insurgency. This structure is revealed by the nature (size, timing, location) of the attacks. What is interesting about the model is that these structures emerge without a centralized control hierarchy. As a final note Robb has this to say about our research paper

As a scientific study of the area, it does a great job.

Update: As Robb notes, in the Nature paper we use a constant population N for the insurgent forces. However, although it is easier to solve a model with a constant population – the structures and mathematical signatures produced by the model do not change if you allow the insurgent force N to increase/decrease over time. I will follow up tomorrow with a post about non-stationary insurgent populations and other generalizations to the model.

One of the important things to understand about our Nature paper on the ‘Ecology of war’ is that the results in this study go beyond first order statistical analysis and power-law distributions. In this paper we look at the deviations from power-law and we analyse the temporal distribution of attacks. This in itself represents a significant advance in the field of quant analysis of conflict.

But perhaps more important still is that we are able to create a robust model to explain the emergence of these statistical patterns. And this model is applicable across a wide range of different conflicts including modern terrorist activity. With this model we can finally start to understand what it is that makes an insurgent ecosystem successful. Going even further, we can run different strategic scenarios on this model to understand how the insurgency is likely to respond to specific policy changes i.e. increasing troop numbers in Afghanistan. We are simply not able to gain this type of understanding with simple statistical analysis – the modeling is key.

Mikael Huss captures this realization nicely in his latest piece

An interesting article about analyzing and modeling data from wars and conflicts was published in Nature yesterday. On first glance, it looked like one of those “look, we found another power law” papers, but after I had read this interview with one of the authors, I changed my mind – it’s really quite interesting.

Huss also notes that the results are independent of the specific source data used for the analysis i.e. the results are not just a signature of the way mass media reports violence.

Even this data-collection effort is interesting in itself. According to Sean Gourley…the statistical pattern remains similar no matter what data source (governmental/academic/mass media) you are looking at.

A new thought provoking piece by David Berreby on the Ecology of war paper is now up on bigthink.com. In the piece, Berreby does a good job of explaining the underlying forces that drive everyday human behavior. He gives examples of simple behaviors such as how we vote, or what we order that are governed by these forces. But he also argues that in large social systems more complex behaviors such as cell phone calls, the flow of traffic or stock market volatility are also potentially predictable – even if the individual people in the system are unaware that their behaviors are essentially being controlled. He notes that this prospect whilst true does not always sit well with our traditional sensibilities,

Those forms of prediction that rubs against our inclination to believe in free will. How could my decision next Friday be forecast by analyzing what millions of other people do? The prospect is a little freaky.

What he goes on to say is that war, like these other large social systems, is also governed by a set of underlying forces. These forces act to define a mathematics of war and can be used to make predictions about when the next attack is likely to be

In other words, according to their model, the decisions of insurgents–about whether to attack on a Wednesday or a Saturday, about whether to try for an average success or go for a spectacularly bloody result–don’t take place in a wildly unpredictable “fog of war.” Instead, they’ll always tend to follow the same rhythm. Regardless of their beliefs, ideologies and motives. Regardless of their immediate tactical concerns. Regardless of what they may think they are doing.

Berreby goes on to explain why insurgencies share these mathematical characteristics, it is a result of group structure, dynamics and decision making.

If there is a common signature to all insurgencies, they argue, it must be because all insurgent fighters converge on the only viable strategy. (The pattern they found in insurgent attacks does not apply to non-insurgent conflicts, they write.) The authors believe guerrilla movements are bound by a mix of physical and social constraints. Physically, insurgent groups maintain a particular size and organization to persist; socially, they have to strike in such a way that they get the maximum media attention and political impact. A terrorist group doesn’t want to strike on a day when three other units also attack, because then their assault will be lost in the general coverage. In other words, guerrillas, like stock brokers, are making decisions based on what they think other people will do.

Insurgents after all are human and like other large scale human systems their actions are now just starting to be understood.

Equation describing the dynamical composition of an insurgency.

If you’ve been following the Nature research and you want more detail about the latest work on the ‘Ecology of war’ head across to the TED blog where I’ve done an in depth Q&A with them. In this piece I talk about distributions, modeling, the role that media plays in covering conflict and the history of the quant analysis of conflict. I also go into the specifics of of insurgent dynamics and look at the four major policy implications of our research.