Potentially insane "Big Data" approach to comp

[bradpowers]

First of all, know that this is *NOT* something that is achievable in the short term, but would make sense in the long term.  There's a significant amount of work that would have to be done to make this happen, but then the system would effectively run itself.  I also acknowledge that this system is probably not perfect, but it's also probably better than nothing.

My idea is to use the fact that some are building large repositories of 40k game data to automatically balance the game (this is similar in concept to how some video game companies balance their stuff).  If we were able to get a copy of every list played in every game in the tournament scene across the country (which is certainly ambitious, but it's already being pursued by folks like Torrent of Fire), we could hypothetically draw conclusions about how competitive a given unit, or even unit upgrade, is.

As an example, consider dumbhammer 40k, where you have 100 points to take exactly one unit.  Then we have a 100 point unit tournament.  At the end, we could compare the effectiveness of each unit.  You can scale this up to even large games if you have enough data, and comprehensive list information (trust me, this is what I do).

So, once we have "effectiveness" data for each and every unit and upgrade in ALL OF 40k, we simply throw out the current points values, and replace them with points values based on effectiveness.  BOOM!  Mostly balanced game. 


A few more notes to curb off pissing and moaning:
- Yes, this approach is hard to implement
- Yes, it ignores things like synergies and buffs and the like (although ultimately this would balance out)
- Yes, it screws with the game at a REALLY fundamental level
- Yes, it's possibly crazy

What I'm proposing we do is, instead of balancing 40k on perceived imbalance, which is a system that is inherently biased, we use what is quite literally the first principal of the game: Paying points to do something, to balance itself.

I've put on my flame resistant suit, so go ahead.
 

[robpro]

I think they might only collect the lists for the top 8, if even that. You really would need every list for something like this to work, but it's certainly possible. House rules like 2+ rerolls being 4+ would make the results not the best, also banning escalation, stronghold assault, and formations stop you from really balancing the format.

[Cryptognomicon]

As a data monkey myself I really love the idea. Two things to take into account. 

1) Luck:  the game is fundamentally based on a players luck. There is math behind each dice roll that you can estimate the potential results on and you would have to take that into account with the calculations.  Not at all saying this is impossible but it potentially complicates things a bit.

2) Player skill:  You could have a terrible player playing a crazy overpowered list.  He/she still looses.  You could have an amazing player who plays a crappy list.  He/She wins.  Things like this "might" throw off the calculations. 

Again, not at all saying that this idea can't be done and I really love the idea myself. 

[Cryptognomicon]

Might be interesting to start on a smaller scale - maybe start collecting data just for BG; lists and player rankings for events.  See how that goes for a while. Then expand to some of the larger events. 

[Cryptognomicon]

Sorry - last post.

One other thing to take into consideration is game type.  Some armies do really well on some game types, while others don't.  That would have to be another data point.  You would really have to break it down by each round I think to make this work.

Round 1: Player A vs Player B playing the Relic Mission, Dawn of war
Round 2: Player A vs Player C playing Purge the Alien, Vanguard

etc

[andalucien]

This is another approach to something I was kicking around a while ago...

Marketplace-driven self-compensating point costs.

For a big tournament like the NOVA you could have the list building process be interactive and take place over the course of a week or so, and the more people that want to run a given unit, the higher the cost of the unit.

This has an advantage over the retroactive, data-crunching approach
- If a new codex comes out (say one with some extremely broken units), you don't have to wait a month or so to start adjusting point values on it
- If something that USED to be good and widely spammed stops being so effective (like due to a GW FAQ or something) you don't have to wait a month or so before you STOP comping it.

You also don't need to actually solve the problem of disassociating units from winning based on synergy with other units, etc.

Of course, huge disadvantage is that you need to write software handling the interactive list registration "bidding game".  (and even deciding how this would work in theory is tricky).
 

[AstartesXXVI]

I don't think you can succeed at this due to the rate at which new devastatingly huge meta changes in the game are being released by GW. However, the core idea of collecting data and drawing relevant conclusions is a good one. One consistent thing I've seen in these discussions over the years is that they always end up with people thinking slightly different things need the nerf bat (and those slight variations are always 1000% obviously related to what army they play).

The other alternative is to not comp at all and simply do more events that use all the different parts of the game separately and make a shitload of money in the process but that is a different animal entirely.

[MM3791]

Well the new (2nd) Tyranid Data Slate might have made them one of the best armies in the game.. from worst to first so to speak. I think formations are a good thing because they allow powerful builds while at the same time they restrict spamming one unit.

[Goblin]

this would work. all of the factors of luck, player skill, etc could be drown out by sheer numbers or could be compensated for with more variables. machine learning is a thing, big data analysis is a thing. the technology is there, but what probably isn't is the data. you need a LOT (it's really hard to emphasis how much ) of data to make this approach work, but i bet if you could get the thousands of gaming communities around the world to pitch in their data, you could actually pull this off. that's a pretty monumental task, and probably not realistic. in theory though, it would definitely work.

[Chase]

I've got lists for days... and results too.

It would take a lot of time to pour through it all, though. 

[bradpowers]

I agree with a number of the points here.  I agree that this is a dice game, and that can skew results, but the point is, if you have enough data, the dice disappear.  Same thing with player skill, and so on.  This also assumes no house rules, this would definitely be something that a significant fraction of TOs would have to be willing to try for it to work. 

Yes, GW releases *stuff* all the time.  I think that for simplicity's sake (given that the number of people playing codex: X is large, but supplement: Y is small), it'd make sense to ban/outlaw supplements, escalation, etc. for these kinds of things. 

As for mission, I completely agree that some units are good at some missions and so on.  Again, if Brad were king of the world, every tournament would have the same N missions, and Brad would know what mission was played at each match.

Player Skill:  Assuming we have vast data, we know player skill pretty well.  Factor that out...

[Benjamin]

The Invitational is as close as we've ever gotten at BG to something like this.

Going unit by unit could fix some things, but some powers in a codex that can swing a game (e.g. Warp Storm table).

With enough information, I feel factors like player skill and luck would eventually curve out appropriately. As it is, the better players rely on luck as little as possible.

[Goblin]

Going unit by unit could fix some things, but some powers in a codex that can swing a game (e.g. Warp Storm table).

this is a random element that, over a large amount of data should either raise or lower the strength of the related units by some quantity, helping to rate those units accordingly. with enough data, unit who benefit more heavily or suffer more heavily from the table should show those effects in the data. again, this comes with the caveat that you need a LOT of data (we're talking on the order of thousands or millions of games) for these effects to show up properly. in theory they should show up, in practice, we just have to accept that the data might skew a little bit with a smaller sample size.

[Benjamin]

But something like the Warp Storm table doesn't always affect units directly, which is why I mention it.

[Goblin]

what i mean is, if it's a powerful thing that assists chaos units more often than it hurts them, then you'll see a natural rise in the statistics for all chaos units. also, for example, if +1 invul is a more significant benefit to some chaos units over others, then with enough data (again, we're talking about monumental amounts of data) those units will end up having higher statistics (even if only by very small amounts). the same goes for enemy units, if some units are more susceptible to the damage results of the storm, or those units dying have a larger effect on the game, then there will be a negative effect on their statistics, even if it is very small (since the percentage of games it will happen will be so low). again, with millions of games your statistics should show the effects of even a random effect like warp storm, even if they are very small. over the course of so many many games, the negative and positive effects should mostly cancel out, meaning that if there is a slightly higher chance of a positive effect, then it will raise the statistics very slightly, or if there is a single very rare effect that has a massive negative impact, it will happen many times (in proper proportion to it's probability) and that effect may or may not outweigh the more frequent but smaller positive effect.

all that said, something that has a direct and consistent effect on games (like say, a riptide) will naturally affect the statistics WAY more and that should push those to the top of the list. this (probably in a ratio to its points cost) would show you which units were 'overpowered' and by how much. now, that doesn't tell you how to fix problems, just which things are problems. ideally, you'd implement a solution and collect more data to prove or disprove it.