The most popular and comprehensive Open Source ECM platform
Let us know more
Artificial Intelligence: Your Miles May Vary When AI Algorithms Go into Production
AI algorithms can be trained. But it’s important to remember that trained algorithms are tuned to work well under very specific conditions using specific training data sets. Trying to apply a trained algorithm on data that has been created or collected in a different way than the original training may not always have good results.
In an interview with Tekla Perry on IEEE Spectrum, Andrew Ng, Stanford Professor and AI pioneer, talked about problems that might arise when AI algorithms are promoted from working on test data in the lab to analyzing data from applications in real life.
Andrew Ng, said “that when we collect data from Stanford Hospital, then we train and test on data from the same hospital, indeed, we can publish papers showing [the algorithms] are comparable to human radiologists in spotting certain conditions. But, it turns out [that when] you take that same model, that same AI system, to an older hospital down the street, with an older machine, and the technician uses a slightly different imaging protocol, that data drifts to cause the performance of AI system to degrade significantly. In contrast, any human radiologist can walk down the street to the older hospital and do just fine. So even though at a moment in time, on a specific data set, we can show this works, the clinical reality is that these models still need a lot of work to reach production.”
Ng said that “all of AI, not just healthcare, has a proof-of-concept-to-production gap. The full cycle of a machine learning project is not just modeling. It is finding the right data, deploying it, monitoring it, feeding data back [into the model], showing safety—doing all the things that need to be done [for a model] to be deployed. [That goes] beyond doing well on the test set, which fortunately or unfortunately is what we in machine learning are great at.”
