Posted by _theprofesso on 5/12/2020 7:22:00 PM (view original):
Thanks all. Gillespie I’m not sure I’m following what your asking. But the data came from seble. I just reached out and he gave it to me in an excel file so I could manipulate it to make these rankings since I am not in every world there is no way I could get the data without his help.
that is awesome seble took the time to get you the data! i'm not sure if you are interesting in keeping these up over time, but it seems like it would be a lot to ask seble to regularly provide updated data. i'm not trying to volunteer you for a bunch of work (ok, that's a lie), but if you were interested in keeping these up, i can help with that. couple things -
1) i fixed my stupid program that does this, i need to spot check it because after the forbidden error was resolved i found the history page changed and broke the program a half dozen ways - but it seems like its working? i'll spot check in an hour when its done, or tomorrow (90% chance it dies before it finishes anyway). i only dump the data in a java only format right now but i can provide a CSV or something you can easily open in excel? my goal would be to get you set up to run the program to scrape the data, so i would be out of the loop. folks who were here a long time ago know how often this gets updated when i have to do something (a couple/few times a year and then nothing for years)
2) you'd probably want to re-check with seble that its cool to run this regularly and let him know you'd be doing it. i don't know i got permission but i did sort of let him know, and then i also ran it at night to make sure the 10,000 requests required to hit all 10,000 team pages didn't slow down regular people. usually i would do this stuff **** faced at 3 in the morning. it runs for at least an hour - although it looks like i took the sleep out to avoid hitting the server too hard. i'll probably put that back in
3) as part of this program it calculates an obscene amount of stuff. it used to take me 30m just to paste it all on the forums for a single run. i suspect you aren't interested in all that but there has got to be something in there worthwhile?
i'm not going to post anything at least for a while, except maybe population data re: that one thread about freehd. my goal for almost a decade has to been to find someone who would upkeep the rankings which is a pretty cool thing for the community (and so that program i wrote wouldn't go to waste), but i am the worst person in the world to carry that responsibility. if this stuff was updated regularly, i kinda think it would be something a bunch of people would come to check regularly for their dynasties. i never did that, but i recall at one point ages ago adding up the forum page views from the ranking threads and it was 10s of thousands. surely would have been much higher if it was updated more frequently!
5/13/2020 1:42 AM (edited)