Hi, I'm Stephanie Stradley, writer at Ultimate Texans part of Chron.com, appreciator of smart statistics, and (distressed) Texans fan.
Today's roundtable on the impact of advanced stats in the NFL includes Cardinals RT Eric Winston, Pro Football Focus writer Sam Monson and Derek Sarley from Iggles Blog & Philly.com.
I'm going to ask a number of questions to moderate this discussion but please, if you are reading this, log in, and feel free to chime in.
First question is more of a living history one. At what point did you become more aware that more detailed statistics other than the traditional ones were becoming more commonly used to evaluate NFL games?
For me, it was around 2006 when I first started writing about the Texans. Back then, the Texans were coming off of a 2-14 season, and most traditional writing about the team was more inclined to be punchline in nature. I started seeking out sites like Football Outsiders that talked about all the teams, and not just the TV popular ones, and gave me additional information that I didn't already know.
I find it interesting that thousands of people can see the same game, but see different things in it. A specific statistic, for me, can start as sort of an objective, neutral starting point for further discussion--whether it confirms what you think, or challenges your thinking.
Over the last couple of years, I've noticed that general public use of statistics is becoming more common in all sports.
I started noticing and following (sporadically) a few years in to my career. More and more people were referencing Pro Football Focus, Football Outsiders, and others. It has definitely added to the level of conversation that the average fan has about pro football.
I took a bit of a strange route I guess. I was just a big football fan. The more of the game I could watch, the better. The stats were interesting, but I wasn't a huge advanced stats guy when I stumbled into Pro Football Focus.
I was lucky enough to know Neil Hornsby through an online football fan site and jumped at the chance to get in and do some tape work studying football games. PFF began as - and still remains at heart - a player performance site. We grind tape and grade players, the stats we collect during that process are essentially just a bonus by-product or something we collect because we're there doing the work anyway.
I guess that was towards the end of 2007 and the start of 2008 when I helped get involved in a site that essentially just featured Neil, Ben Stockwell and our IT guy Ian Perks at the time.
The interesting thing now I think as statistics get ever more specific and in depth is trying to identify which stats actually mean something, or what exactly they mean, and when they can lie to you.
The bottom line with all of this though is that we're just trying to go deeper than the game was being covered before.
I just checked the archives on my old blog and it looks like the first mention of Football Outsiders also came in 2006 on a post that -- funny enough -- started out talking about the Texans (http://www.igglesblog.com/i...). So that probably counts as my official starting point with advanced statistics as well.
While it's always fun to look back, I'm more curious about where people think we should be going next.
I've never been to the Sloan Conference, but it seems to me there have been two major waves in advanced NFL statistics. Wave #1 was led by Football Outsiders. It was very equation and model driven, with the underlying assumption being that if we just built a good enough regression model, we'd be better able to predict things like game outcomes, draft success and free agent impacts.
The second wave, which we're in right now, mostly eschews the advanced mathematics and instead relies on the greatest invention since the transistor -- the all-22 coaches film. This new (for fans) view of the game is helping to power a rapid evolution in our understanding of schemes and player responsibilities, which is feeding directly into the kind of work done by site like PFF.
The irony of the current model is that all this exciting Wave #2 technology is being used to generate Wave #0-level statistics. There's nothing in the coverage stats that accounts for the pass rush (or lack thereof). There's similarly nothing in the pass rush stats that accounts for the coverage. A QB hit is a QB hit is a QB hit.
It's actually nice that Sam said the stats PFF publishes are just a byproduct of their grading work. It would have sounded a lot more critical if I'd been the one who said it.
For starters, the PFF folks would, I'm guessing, argue that their grades are the tool that take into account all those complications I mentioned above. A rusher who blows past the OT and hits a quarterback in 1.2 seconds will get more credit than one who mills around for awhile before falling into a lucky hit when the QB blindly fades his way.
But the problems with the grades are legion. For starters, they're a black box. I re-watch every play of every Eagles game multiple times, and it's impossible for me to check to see if a blown coverage has been assigned correctly by PFF's graders. I can try to back out the numbers for a single game based on the "thrown at / completions" stats, but that's not a scalable effort.
The second issue is that single-number (or even four-factor) grades are an incredibly blunt tool. This guy is 14.1 and that guy is 11.2, so clearly the first is better. Well, maybe. Or maybe the first is given easier jobs because he has better teammates or his coaches know they need to protect him.
This is where NFL analysis -- in my mind -- is clearly behind the other major sports. Let's say, for the sake of argument, that PFF's grades are the best individual metrics we have. Using Eric Winston as an example, that means we "know" the 2013 season looks like this:
The bad news is that it will be a long, long time -- if ever -- before this is information the fans can get their hands on. But that doesn't mean someone couldn't get started by, say, tracking every route receivers run by hand. After all, there are already people re-watching every play of every game.
Advanced stats in football have exploded over the past few seasons, but still don't dominate the conversation the way they do in baseball, and to a lesser extent, basketball. Is Football going to become more stat heavy? Are players aware of advanced stats? Some more than others?
I would indeed always point to PFF's grades over anything else. Those are the numbers that put the intelligence into the statistics. All sacks look the same on the stat sheet (well, aside from half-sacks, but I'll gloss over that particular ridiculous quirk of the NFL), but did it come from Robert Quinn destroying the OT to turn the corner in 1.5 seconds and drag the QB to the ground (be thankful you're a RT Eric! You 'only' have to handle Chris Long...), or did the sack come because the QB got happy feet in the pocket, danced his way into trouble and then hit the deck for someone to eventually touch him 5 seconds into the play?
The interesting thing I think is that PFF's database has that kind of data in it, Derek, and we actually offer a lot more than goes up on the site to teams behind the scenes for let's just say a significantly larger sum than $26.99 a year.
I think the use of data is definitely getting more sophisticated, within PFF and everywhere else. Our issue at the moment is that we're working from an IT model that's a few years old - we haven't been able to keep up with our own growth and the demand. Hopefully that's something we'll begin to catch up with in the future. You'd be very excited if you saw some of the stuff we have behind the scenes.
As for whether players are aware of them - players are certainly aware of their PFF grades judging by my inbox each week. I get players emailing looking for an extra stat here, or a stat taken away there. Guys are getting their grades put to them in locker rooms and asked to justify it or react to the praise etc. They definitely care, more than most people give them credit for I think.
But Eric I'm sure has a better insight on that than I do.
I'm at the risk there of sounding like a walking advertising board!
I think another interesting area is what the teams are now doing as they get deeper into stats and analytics. Teams are often very reluctant to spend money, but they're investing heavily on new Analytics Departments because they see the competitive advantage it can bring.
Jk, kind of. (If they are, let me know, need to have a kind word with the people at PFF) :). I don't think the majority of players look at advanced stats but there are some that definitely do.
Derek brings up a good point about grading. I honestly don't know what my PFF grades are this year. I do know however in the Seattle game, PFF had me down for 8 or 9 hurries I believe but my coaches had me down for 2 or 3. Well let's look at the first problem: what is a "hurry". Is it forcing the QB to step up because of short corner? Is it a forced throw? Is it moving the QB of his "spot"? My point being advanced stats get into murky waters due to the lack of universal definitions.
On another point. I don't know if football will ever be at the same level as other sports. This is the ultimate team game for a reason. If a WR runs the wrong route, that directly effects my performance. That's just one example out of literally the 1000s that you could easily think of right off the top of your head. With baseball and basketball (to lesser degree) there aren't nearly the variables and possibilities.
While I think it is awesome PFF and other sites have people more and more involved, fans hold these stats to be 100% true and there is no wavering from it. I just don't think you can grade an Olineman (for example) without being in the meetings and especially knowing what the play call is and whose responsibilities everyone really has.
And I apologize in advance. My typing skills are the worst in the league. I can't type fast so I tend to leave out words and jump from topic to topic so just hang with me. Also if it wasn't for autocorrect it would look like I failed several spelling tests.
As to Eric's point, I think it is also extremely difficult to separate individual play and talent from scheme/coaching/teamwork. A offensive line's performance as a whole can be vastly affected by who the quarterback is. Or whether the scheme tends to be more difficult or easy for a lineman to hold a block.
When evaluating players for free agency, those sorts of factors that show up when you watch a team but may not show up in grading can be very important.
I think we've all come across the big issue with football - everything is so interlinked and related. How do you separate a running back from his blocking with stats? Even PFF grades need to be used with thought and context to them. Two tackles can allow the same amount of pressure but one has a quarterback averaging a league-low time to throw average and the other is at the high end of that scale. Or one of those guys may have faced a who's who of pass rushers.
I don't think we'll ever be in a situation where stats can replace watching tape, but we can certainly augment it.
I also agree that no set definitions of many of the stats leads to a lot of confusion. So much is subjective when it comes to football, definitions change team to team let alone stat service to stat service.
The issue of context also goes well beyond the micro-level issues Eric identified. For example, Eagles WR Riley Cooper played six games with Mike Vick as his starting quarterback this year. Here were his grades:
-0.6 -0.2 -2.7 -0.1 -1.2 -1.3 --------- -6.1
He also played seven games with Nick Foles as the starter:
1.7 2.0 2.2 1.9 0.5 0.7 2.1 --------- +11.1
I'm not sure I could find a better illustration that what PFF is grading is not "Riley Cooper" but rather "Riley CooperWithVick" and "Riley CooperWithFoles." Much like, and not to put an active player on the spot, it's "Eric Winston in his old context" vs. "Eric Winston in the new one."
This is why I think the second wave of "advanced stats" is heading towards a dead end as it's currently being done. It's good for fun arguments about which guy should be in the Pro Bowl, but it emphatically does not settle the debate of which player is a priori better in the way that so many people think it can (with perhaps a little tweaking).
I'd hasten to add that the first wave of advanced stats ran into the same types of roadblocks for the same reasons. It's very, very cool to model team performance in a way that accounts for lots of different variables and maybe tells us more than the records or regular team stats do. But the path ran into a ditch with individual projection models like the Lewin Career Forecast and sackSEER.
Getting back to a couple of the links I posted above, how freaking cool would it be to see spatial tracking of offensive linemen to see how much push they got on run plays inside or how far they were asked to pull outside? Or every route a receiver ran so we could visually understand the difference between a guy like Wes Welker and one like Victor Cruz?
In those cases, the data gathering happens first and then the analysis comes at the back end, once we have it, rather than the decision being made right up front a la "was this a successful play?"
Part of the issue I think is that PFF has no control over how people use PFF data. I know initially Neil was dead against even having overall rankings, because people would just look at them 1-50 and say "player x is 20th, player y is 25th, so player x is better".
PFF isn't trying to evaluate talent, or skill, or greatness or any of that but rather performance or efficiency. Riley Cooper is a more efficient receiver with Foles at qb than with Vick.
I do wonder at times whether we are swimming up river in terms of presenting something other than what people are looking for or assume we're presenting.
Eric just hit on the biggest confusion with PFF's model: grading is not an advanced stat, and advanced stats are not grades. They shouldn't be conflated.
Grading is a subjective performance evaluation, as Sam describes above. Certainly it's informed by statistics and close film watching, but different viewers can have different opinions, and a lot of inside information is necessary to completely understand blocking assignments, coverage responsibilities, and the like.
PFF also gathers a fantastic collection of "real stats," as Eric labeled them. Charting plays at a more granular level, applying snap count data, and providing it to the public is a huge advance in NFL advanced statistics, building on work that others like Football Outsiders have done. There's still room for error and questions of standards, but PFF's Pass Rush Productivity is a great example of an explanatory stat that goes beyond sack total.
But those stats aren't grades, they're stats. They don't have narrative value until you put them in context and don't allege to provide be-all, end-all analysis. Unfortunately, PFF's emphasis on the latter conflates the two areas.
Like anything in the league some teams are more open about it than others, and some teams really like to keep it quiet believing it's a competitive advantage.
We sell various packages to teams and we're constantly having teams come to us asking us if we could collect a new piece of information or other, but very few of them want to let anybody know what they're doing on the inside when it comes to metrics and analytics.
It is interesting though how teams are now almost being forced to openly discuss analytics in the general course of their day to day interaction with the press. Justifying calls or no-calls on 4th down, reacting to performance of players put to them in advanced stat form etc. Coaches and GMs are being confronted with more information than anybody had in the past and I think you can see it often unnerves them.
I've talked to some NFL people about using numbers to evaluate differences between referee crews. Mark Cuban was very up front about that in the basketball world. In the NFL, I've heard that some teams very much focus on that and give reports to players, and some teams don't. (Heard Belichick tree cares about referee tendencies). Seems to me that in a game where a call here or there can be a huge difference maker, knowing crew tendencies and passing that along to players in a practical form would be helpful. Thoughts?
That's a depressingly good idea, particularly at positions where players can choose to be a bit more aggressive or back off (CBs especially).
The Eagles have been involved in a couple games recently where the officiating was perhaps a bit off-median. Some of the defenders talked after the games about how they adjust to how the game is (or is not) being called. Ref scouting gives a heads up on that.
Having said that, I think there's a larger point here. Fans have always complained about the refs, but in the last couple years it's reached a fever pitch of whining.
I think part of this is because of changes in the rule book that have made the game harder to call. But it's a broader trend in all sports, pushed by the vanishing room for ambiguity left by ever better broadcast technology.
What I'd like to see is someone do an analysis of the real impact of officiating over the course of a season. Brian Burke (http://www.advancednflstats...) already has databases that log the impact of every single play in terms of shifts in expected points or winning percentage. I'd love to see him -- or someone else -- run every called penalty through that system to see a) what the total impact was leaguewide and b) how teams were differentially affected.
It wouldn't solve every argument, but at least the right fans could whine about being screwed while the other 80% could stop yapping about it.
Eric brings up a great point about the hurry stat. I was wondering if PFF gives a hurry when ever an offensive lineman allows a defender to get close enough to disrupt a QB, regardless if the QB is about to throw the ball or not.
For example if a QB three-step-drops 1, 2, 3, then throws, and a tackle stays between his man and the QB until a split second before the ball is out, is that a hurry even though the ball was coming out on that timing anyway?