Researchers are now proving repeatedly that while the black box engines may be inscrutable, they aren’t above being reverse-engineered
By Siddharth Pai
India’s patent laws allow for reverse-engineering of certain technologies. A prime example of this reverse-engineering is in the pharmaceutical space, where Indian pharma companies are allowed to reverse-engineer drugs, especially life-saving ones. These drugs may have been developed by pharma majors in other parts of the world—and then introduced into western markets—after India-based outsourcing firms had helped them out with clinical trials, data gathering and reporting to the US Food and Drug Administration (FDA) or its equivalent to get these drugs passed.
Indian courts have continued to allow such reverse-engineering of drugs—famously prompting Bayer AG’s then CEO Martin Dekkers to say at a conference a few years ago, “We did not develop this medicine for Indians. We developed it for western patients who can afford it.” He called this sort of duplication “essentially theft”.
Dekkers’s outburst was in response to India’s Natco Pharma winning a court case that allowed it to produce and sell Bayer’s Nexavar, an anti-cancer drug, at 97% less than its original cost. Currently, the drug costs $96,000 a year in western markets, but even at 3% of that, it sells for around $2,900 (or almost Rs2 lakh) per year, still a considerable sum for many Indians.
Dekkers’s candid comments give lie to the large drug houses who claim that they care about global health needs since it makes clear that they actively seek to keep life-saving drugs out of the hands of most of the world’s population who cannot pay the insurance-inflated prices they demand out of patients in the West. His outburst also begs the question: If all Bayer did was to develop the drug for westerners and not for Indians, then why should he care if the drug is being sold in India at a fraction of the cost? It isn’t as if the FDA is about to allow a version produced in India to be imported and sold in the US while his drug is still under patent.
This isn’t a rant about western drug companies, though I suspect I have made a point. It is actually about another type of technology that could cost Indians—and Germans, Americans, Britons and others—their jobs. This technology is the machine learning and neural network part of artificial intelligence or AI, where inbuilt algorithmic engines allow computer programs to reprogram themselves without human intervention, after consistent use, into ever better predictors of outcomes such as image recognition. These engines are often referred to as ‘black box’ systems by computer scientists, referring to the fact that it is very difficult for even computer scientists to themselves predict what the engine will be capable of doing after it has rewritten itself several times over using machine learning without human intervention.
A recent edition of Wired magazine, however, points out that researchers are now proving repeatedly that while the black box engines may be inscrutable, they aren’t above being reverse-engineered like drugs (or “stolen” to mimic Dekkers’s terminology). Wired reports that a team of computer scientists at Cornell Tech in New York, the Swiss Institute Ecole Polytechnique Federale de Lausanne and the University of North Carolina recently published a paper titled ‘Stealing Machine Learning Models via Prediction APIs’. APIs or application programming interfaces are built into a computer application to allow programmers and other computer applications to access it. What these researchers have found is a way to create their own artificially intelligent interface into a black box and then use the output from the box to reconstruct its internal workings, thereby reverse-engineering the box.
This is mind-boggling. According to Wired, apart from being able to reconstruct the pay-per-query machine learning engines developed by AI leaders such as Google, Microsoft, Amazon and IBM, they can in some cases also recreate the private data that an AI engine has been trained with. This artificial recreation of data is something that today’s data privacy laws in the European Union or the US simply cannot deal with, and there is no doubt that there will be more legislation in the offing—all of which, given the latency between what happens in the real world and legislators’ response to it, will likely just be an attempt to lock the stable door long after the horse has bolted.
And in an even more sinister revelation, other researchers have shown that they can even learn how to trick the original black box. According to Google researcher Alexey Kurakin, one can slightly alter images fed to image recognition black boxes so that the machine learning/neural networks see something that isn’t there. Evidently, just by altering a few pixels in an image, which are imperceptible to an ordinarily intelligent human eye, an artificially intelligent program can be fooled into thinking that an elephant is actually a car!
Making predictions about the future of technology is a mug’s game—though I am often lured by the turn of a year into trying to doing so. I do prefer, however, to try to use history as a harbinger to the future. One thing is certain about 2016. It proved to be a year where ‘Big Data’ had spectacular failures. Pollsters, statisticians and their intelligent computers wrongly predicted the outcome of both the US presidential election and the Brexit vote. Maybe 2017 will be the year where we see much of the hype around AI being shot down by methods such as reverse engineering—or better yet—fooling the black box engines into arriving at the same wrong conclusions as the Big Data boys did.
Siddharth Pai is a world-renowned technology consultant who has personally led over $20 billion in complex, first-of-a-kind outsourcing transactions. [LiveMint]