If you have been following this baseball analytics segment of my website, you know that I had purchased the Out of the Park Baseball (OOTP) simulator for the 2019 season. The simulator allows users to manage a team (or teams) and play out full seasons. These teams can come from Major League Baseball or several other leagues that are included with the software.
I bought the game to have fun with it as I love baseball. But one feature I discovered that intrigued me is the live daily updates of players and teams stats. This is the first time the creators of the software (or any in this class of softwares) have included this feature.
From a data science perspective this live update feature gave me the opportunity to track how data is used in predicting baseball. The company has simulators for other sports, too. But, I decided to concentrate on just baseball.
An Artificial Intelligence Engine
The creators of OOTP advertise that the software contains an artficial intelligence (AI) engine. That is another reason for covering the software as that topic fits well in the data science arena.
In machine learning, which is a subset of AI, algorithms use actual results to measure against predicted results to see how well it did in predictions. Then, it adjusts and tries to close the gap between any wrong predictions and the actuals results. This is part of the "learning" aspect by the machines.
An AI engine can be programmed to take it a step further.
It can use a neural network with a concept known as backpropagation, which you can think of as a do over with new and better data after each run (there can be millions of runs). This is obviously an oversimplification, but hopefully, you get the idea. The algorithm corrects or learns based on the information it is given subsequently. In general the more data it has, the better. But, the neural network is at liberty to pick better data, rather than simply rely on random samples.
AI engines can do so much more than this simplified example. This is just to explain the concept in as simple a format without getting into the gory details.
I include these explanations because it serves as the foundation for my reasons why I postponed my simulations with OOTP.
I Ran Daily Simulations
Back in March 2019, I decided to experiement with the software to see how it did at making predictions. I bought a previous version of the software last year but that version did not contain live updates. Therefore, the predictions from the previous version would have been made on simulated runs that would not adjust as the season progressed or at least, they would adjust based on simulated runs.
If you are familiar with AI engines, they adjust their algorthims when new data is presented. They learn, if you will. The live updates would be the data the AI engine could use to improve.
I ran the simulations daily and observed the results. I wrote about most of my findings on this website, at least most of April. I kept running the simulations, but stopped writing about my findings (I still have the spreadsheets for the rest of April and the first 10 days of May). The truth is, the predictions weren't improving much. I had a run rate of about 60% games predicted correctly.
To understand the assumptions I used to run the simulations, read throught the starting post for this experiment.
Artificial Intelligence (AI) Engines Should Learn and Improve
I don't claim to be an expert in AI. I have been learning it and my skills are improving. I do understand the high-level concepts and I am starting to dig into the technologies behind it. One underlying fact in AI is that the engines are supposed to get better as they learn. Otherwise, what's the point in companies investing serious time and money into this technology?
If a technology doesn't pan out with whatever promise it makes, then it won't be adopted by society at large. Therefore, I am well within my right to conclude that AI engines should improve as they are used more. This is a fundamental aspect of the discipline.
The problem I had with OOTP, and the reason why I stopped running the simulations for the purposes of experimenting and observing is that the engine is not improving. That 60% correct prediction rate is constant throughout April and part of May.
As I have explained in previous posts in this baseball segment, April should be viewed as a warmup as teams are finding their groove. But, by May you would think the teams would have found their groove.
The OOTP Software Has Not Found ITS Groove
It's not the teams that are the problem, however. It appears the prediction engine is stuck in that 60% zone. I can't say with certainty whether 60% is a good or bad predict rate. It probably depends on how people use it. Someone who is a casual observer may think that rate of prediction is fantastic. For a gambler though, it means 40% potential losses. That's not likely to bode well for them.
OOTP simply did not improve and it made me lose confidence in its ability to be a valid prediction engine, at least one that can carry the distinction of an AI engine which is supposed to improve.
Back on April 19, I decided to do a recap up to that point on April simulations. As you'll read in that post, I recognized that it was early in the season and that should be a factor in the simulator. I did state then, the fact that AI engines should be capable of learning.
Where Do We Go From Here?
The main reason I stopped running the simulations is that they took too much time to process. It required gather the team schedules from MLB.com and formatting in my spreadsheet template. Then, it required running each game and recording the results manually. I set the spreadsheet template up to display which teams were predicted to win for that day. Then, the following day, I would head back to MLB.com and record the results of the game, again a manual process.
After gathering the data, I had to write about my findings for that day. To be honest, it got a bit tiresome for me and felt that readers would feel the same. If results are the same (~60% predict rate) day after day, that isn't newsworthy in my opinion. And, due to the effort required to run and record these simulations, I just felt it was no longer a worthy activity.
July 7, 2019
I decided to take another stab at running the simulation process for the July 7 games. I was thinking perhaps the passage of time would allow the simulator to find its way. I gathered all my data, ran the simulations and recorded everything on my spreadsheet template for July 7. I tried to hold back guessing how the predictions would go but I couldn't help think deep down that it was going to come in around 60% just as it had in the past.
After the entering the scores in my template, it confirmed my suspicions. The prediction rate was exactly 60%. Here we are in July, past the halfway point of the baseball season, and there is zero (0) improvement in the results!
Midseason Recap
The OOTP creators run simulations at the beginning of each season to predict who will be the winners in each division at the end of the seasons. They run 1000 games and record a video on their analysis of the picks. You can find the video for 2019 here:
The video was released right before the season was set to begin. It is an interesting watch. But, I wanted to do a midseason recap to see where these predictions are on track. After analyzing their picks, they predicted that the World Series contenders would be the Houston Astros vs. Washington Nationals, with Washington winning the World Series.
I created another spreadsheet that took the regular season predictions and lined them up with the actual results up to this point in time. As you can see from the image, the Yankees and the Dodgers are the two standout teams thus far. They are both bolded and in red text. They have the higest overall percentage (PCT) in their respective leagues.
Data for predictions courtesy of OOTPDevelopments. Data for Actual Standings courtesy of MLB.com.
A few caveats: there are still many games to play and a lot can happen in three months, especially in baseball. I get that. Also, this analysis is crude and doesn't take into account wildcards, etc. The image shows a snapshot of the standands as compared to the preseaons picks from the software. You can interpret this however you like. I just provide it as a point of reference.
It's interesting to note that one commenter in the YouTube video (who made the comment at the time of the video's release three months ago) stated that he felt the OOTP picks for the World Series were incorrect. He stated the Yankees and the Dodgers would play against each other and the Yankees would win the series. The two teams in red text in the spreadsheet above are the Yankees and the Dodgers.
As you can see, Jason Parker (Commenter in YouTube) is spot on with his prediction. Caveats aside, good job Jason!
I have decided to continue with the simulations, although I won't run them daily. I don't see the point. If during future runs I see a wicked improvement in the predictions, I could change my mind. Hey, perhaps the progession is exponential? Who knows?
Be Careful With Vendors' Use of the AI Label
I am not here to bash the creators of OOTP. I think it's fantastic software and is fun for people who love to dig in and be part of the action. My motivation for this experiment was from a data science perspective, not necessarily from a sport perspective. Don't get me wrong. I love the game of baseball. But, the sport has been intertwined with statistics since the start of the game. It has its place on this data science website. Henry Chadwick is believed to be the father of statistical analysis in sports and started tallying basic stats in 1859.
[Source: https://www.liveabout.com/baseball-statistics-glossary-321381]
I have no way of knowing how the engine in OOTP was developed. Whether you can call it an AI engine is not up to me. The creators believe it is an AI engine. Great. But, how come it is not improving which is a cornerstone property of AI engines? That leaves me with skepticism.
I don't expect any engine to be right 100% of the time, even AI engines. However, when companies label their solutions as AI, those solutions should be much better than humans at predicting. They should also be much better than other software solutions that predict without AI at their core. After all, if we are going to let robots take over the world because they are better than humans at everything (according to "experts"), they should be expected to do a much better job. Period.
Perhaps I am not using the software correctly. But, I have not found any instructions that show how to run simulations. And, to be honest, I don't have the time to chase this down. Any company that makes a claim that it has an AI engine should show people how to use the engine properly. As a former software developer myself, if you don't want people to use your software in a certain manner, you need to prevent them from doing that in the software. So, for all intents and purposes, what I have been doing is correct, unless someone comes forth and tells me otherwise. The software, after all, let me run it in a manner that I believe is correct.
While the concepts of AI have existed for a while now, implementation is starting to take hold. People are both skeptical and awed by what they are already seeing, and we are just getting started. To get people to adopt AI, solutions providers must take the lead and showcase how AI will help people. Otherwise, society society will continue to be mired in dystopian visions of robots taking over peoples' lives.
At the other end of the spectrum, solutions providers will likely flaunt the AI label to boost sales. Whether they truly have AI solutions is anybody's guess. These black box solutions keeps an air of mystery, and let's face it, mystery sells.
If you think I am being overly dramatic for a baseball software simulator, I can assure you I am not. Sportscasters are referencing the predictions made by OOTP. It's getting attention, so it is a big deal.
I welcome comments by anyone, including the creators of OOTP. I hope that this analysis is taken in the spirit for which it was intended, i.e., to address the issue of AI engines improving over time and why OOTP hasn't changed much through the season.
For pure entertainment purposes, I recommend buying OOTP. If you are thinking of purchasing because of its AI engine classification, it may be wise to proceed with caution.
