The risks of over-reliance on quantifiable data

By: Jan Kallberg  

Source Link

The rise of interest in artificial intelligence and machine learning has a flip side. It might not be so smart if we fail to design the methods correctly. A question out there — can we compress the reality into measurable numbers? Artificial Intelligence relies on what can be measured and quantified, risking an over-reliance on measurable knowledge. The problem with many other technical problems is that it all ends with humans that design and assess according to their own perceived reality. The designers’ bias, perceived reality, weltanschauung, and outlook — everything goes into the design. The limitations are not on the machine side; the humans are far more limiting. Even if the machines learn from a point forward, it is still a human that stake out the starting point and the initial landscape.

Quantifiable data has historically served America well; it was a part of the American boom after World War II when America was one of the first countries that took a scientific look on how to improve, streamline and increase production utilizing fewer resources and manpower.

Numbers have also misled. Vietnam-era Secretary of Defense Robert McNamara used the numbers to tell how to win the Vietnam War, which clearly indicated how to reach a decisive military victory — according to the numbers.

In a post-Vietnam book titled “The War Managers,” retired Army general Donald Kinnard visualized the almost bizarre world of seeking to fight the war through quantification and statistics. Kinnard, who later taught at the National Defense University, surveyed fellow generals that had served in Vietnam about the actual support for these methods. These generals considered the concept of assessing the progress in the war by body counts as useless; only two percent of the surveyed generals saw any value in this practice.

Why were the Americans counting bodies? It is likely because it was quantifiable and measurable. It is a common error in research design to seek the variables that produce easily accessible quantifiable results, and McNamara was at that time almost obsessed with numbers and the predictive power of numbers. McNamara was not the only one.

In 1939, the Nazi-German foreign minister Ribbentrop, together with the German High Command, studied and measured the French and British war preparations and ability to mobilize. The Germans quantified assessment was that the Allies were unable to engage in a full-scale war on short notice and the Germans believed that the numbers were identical with the factual reality — the Allies would not go to war over Poland because they were not ready nor able. So Germany invaded Poland on the 1st of September 1939 and started WWII.

The quantifiable assessment was correct and lead to Dunkirk, but the grander assessment was off and underestimated the British and French will to take on the fight, which led to at least 50 million dead, half of Europe behind the Soviet Iron Curtain and the destruction of their own regime. Britain’s willingness to fight to the end, their ability to convince the U.S. to provide resources, and the subsequent events were never captured in the data. The German quantified assessment was a snapshot of the British and French war preparations in the summer of 1939 — nothing else.

Artificial intelligence depends upon the numbers we feed it. The potential failure is hidden in selecting, assessing, designing and extracting the numbers to feed artificial intelligence. The risk for grave errors in decision-making, escalation, and avoidable human suffering and destruction, is embedded in our future use of artificial intelligence if we do not pay attention to the data that feed the algorithms. The data collection and aggregation is the weakest link in the future of machine-supported decision-making.