CometML's ML Practitioner Survey on MLOps
Created on February 24|Last edited on February 24
Comment
CometML just released an ML Practitioner Survey, surveying 500 practitioners about the state of MLOps and standing issues. The current MLOps conclusions are listed below.
Summary
- 82% of practitioners say it's difficult to evaluate the risks and limitations of models
- 28% say their company's ML budget will decrease in the next 12 months
- 41% say process sustainability is one of their biggest challenges
- 42% say they are using a commercial experiment managing tool
- 27% believe that bias will never be removed from AI systems
- From the sample, on average, 41% of ML experiments had to be scrapped due to
- API integration errors
- lack of resources
- inaccurate/mismanaged data
- manual mismanagement
- On average, the time to deploy an ML project is 7 months, a 2 month increase from last year
Anticipated Challenges in 2023
- Explainable AI: 36%
- Hiring for Institutional Knowledge: 36%
- Employee retention: 39%
- Process sustainability: 41%
Biggest Obstacles for Practitioners
- Resources: 24%
- Reproducibility: 26%
- Infrastruture: 27%
Most common ML tools for tracking ML project development
- E2E platform: 38%
- Google Sheets: 38%
- Source experiment tracking tools: 39%
- Commercial experiment management tools: 42%
Most Common Economic Impacts to their Company:
- Little to no MLOps budget: 35%
- Hiring Freeze: 36%
- Impact on budget: 37%
- Redundancies in teams: 40%
Conclusion
They conclude with their survey results on AI Bias and AI Bill of Rights published by the US White House Office of Science and Technology Policy (WHOSTP).
- 73% of ML practitioners believe an AI Bill of Rights (BOR) is necessary
Pros
- 38% believe BOR will make it safer
- 37% believe BOR will reduce privacy violations
- 35% believe it will reduce the frequency of unsafe ML systems
Cons
- ~40% believe BOR will lengthen the ML deployment cycle
- ~35% believe it will make deployment more expensive
Regulating the development and deployment of AI is a difficult task. Not just for the engineers involved in the business, but also for maintaining and controlling its black box nature (especially for more advanced DL applications). Difficulties lie in not just mitigating bias but also developing the actual tools and infrastructure.
References
Add a comment