Fixed a few bugs
Bowl game status now shows up
Separated out wins/losses into separate columns for the coach scraper
So I had an epiphany earlier today and figured out how to make this thing run way faster. Multi-threading! There is now a dropdown to select the number of threads you want the program to use. More threads means more simultaneous page requests, but it will also take up more of your computer's resources. You'll have to test a few values to see what number of threads works well for you. My computer (dual core) seems to fly when I set it to 20, but it actually gets a little slower when I set it to 24. If anyone is rocking a quad core, they could probably handle 24 just fine. I can now grab the entire coach pull for a world in about a minute. I haven't done time trials for comparison, but I'm pretty sure that's over 10 times faster!
I've updated the link in the post. Grab the new version and experience the speed. And yes, I will be working on using multi-threading in my other tools now
I had made a few command line scraping tools awhile back for bhazlewood. There's been some recent interest in them, but the fact that they're command line only has been a barrier for some. So I packaged them up into a GUI. You can open this one by double clicking on it (just like all my other tools). In order for it to work, you HAVE to have the permTeams.dat file in the same folder as it. This is the same permTeams.dat file that's used for the recruiting tool.
Right now, the scraper utility only includes my roster scraper (gets team rosters, form IQs, GPAs, and season stats) and my coach scraper (gets pretty much everything off the history page for each school). There's a tab for a game scraper which I'll be including later on.
Necessary file that goes with it (you may already have this file if you use the recruiting tool):
(you'll have to unzip this to get the permTeams.dat file)
As always, please let me know if you find any issues with it.
2/19/2012 3:38 PM (edited)