We hear this question frequently and it makes sense — when you train a machine learning model, you probably want to know that it's better than what you're currently doing. You probably also want to know whether it's worth the investment.
But whether or not it's worth it heavily depends on your situation. Let's consider a binary classifier model — say, a model that tells you the probability of a lead becoming a customer (or not). If you flipped a coin, it would be correct 50% of the time.
When predicting things like human behavior, our customers have success with models typically between 60-80% accuracy — meaning, the percent of predictions the model will make correctly.
Suppose you've got 4,000 leads every single month. A model that's only 60% accurate might mean a massive improvement in how your sales team can target. Suppose each rep can speak with 30 people per week and you have 10 reps. They can only speak to 1,200 leads. By improving their ability to target the right leads, even just a little bit, they can move the needle quite a bit for their business.
Compare that to a business with only 150 leads per month and 2 reps, who can also on speak to 30 people per week. With the capacity to speak with only 60 leads, they really need to get it right. Maybe they need a model that's 80% accurate to get the results they need.
When you think about predicting a continuous variable, say, the amount of orders that a shipping company will receive next week, it's a similar idea. They need to have the right resource allocation: Order too much, they're overstocked. Order too little, they're out of stock. Their profit margin might allow them a certain rate of error here. One business might be able to afford a 25% error margin in their orders, another business might need to keep their error under 1%.
In short: Understand your business to calculate your ROI. We're more than happy to hop on a call and talk through it.