Last week, I sat down with Panda Global's own Suar as the rankings for the fourth season of Smash 4 competitive tournaments were being published. His team, "PGStats", are solely responsible for creating the official bi-yearly rankings.
As the season closed with the last event in the middle of December, the team has been hard at work making sure nothing is out of place for the announcement of the current top 50 Smash 4 players in the world. As the rankings were being published throughout the week, I ask Suar about some more technical questions revolving around creating the rankings and how he finds success in digital media in our current climate.
This interview has been editted slightly for readability and answers are shorted to just specifically what was asked. If you are interesting in all that Suar had to say, you can view the full interview here.
Lance: Thanks so much for taking the time to get on with me.
Suar: Yeah, my pleasure. How've you been?
I'm doing good, how are you?
Great. Middle of a release, but we are still kicking.
Is the data you and your team collect automated or do you have people on the team who record the data by hand?
Haha, that's all automated. I think we went through 10,000 pieces of information that were extraneous down to the last 1,000 bits. Stuff like set counts and stuff. You have probably been seeing it in the releases, everyone that is featured has their wins record against everyone they competed against and lost to. Lots of things are not brought up there. We tended to feature top 32, but everything out of that is automated with our database thanks to Giant, one of the newest PGStats members, and PracticalTAS who makes the algorithm & he has been with us for 2 seasons now. We haven't hand recorded anything since the first season.
What would you say is the hardest part of creating a new season's rankings?
Hilariously enough, the rankings are the easiest part. They have been done since December, two or three nights after the 2GG Championship. All of our data was updated at that point as I had my segment there at the 2GGC finals. The rankings aren't difficult because it's just a couple of Excel spreadsheets that calculate everything. The biggest hurdle is the video production. We have to condense the information into a format that video production can understand. If you have seen the videos, there are tons of information featured for every character. Main, secondary, name, region, rankings from other PGRs, etc. Immediately it gets to be a lot. For every 10 people, there is about 1,000 pieces of information that quickly adds up. You have everyone they win or lose against, the character in the match, nationality. Giving all of this information to someone to make a video is a game of telephone and is how our typos get created. The hardest thing every season bar none is the video production. People take the production for granted heavily, even I have.
What are some of the different approaches you guys used when creating the list and attempting to make it as widely seen/successful?
It depends on what you define as "success". Being frank, the releases themselves is more so done as a proof of concept repeatedly for sponsors and other groups to understand how this happens. We even did a mini PGR for Mortal Kombat X. The thing about SMASH however, unlike the FGC, is that it isn't fragmented. The FGC isn't all on reddit, smashboards, or twitter. Smash hangs around on twitter and reddit a lot. You have two groups that you instantly have feedback and exposure to. This is one of the reasons we have some of the most widely shared content started with the platform. Panda Global taking the risk on endorsing the rankings and giving us a platform to be successful was huge. We always put ourselves out there so people know we aren't doing different things in the background and controlling things [...] We take in feedback all the time. I always tell people to tell me when they dislike it because it's just as good, if not better than people who tell me who enjoy it. Having the support of certain community figures also helps. We had ESAM voice over the first season and that led to some people thinking it was his opinionated list, but he was there for his opinion to make it more real. Community leaders accept it and also bounce it between each other. It's salient to every conversation. When people get signed, people always say PGR tournament, PGR set count, PGR ranked player. It's now people's goal to get on PGR or being a ranked player. That has also helped because, when people produce some things, they don't stick because they aren't specifically important for the scene. It's more of a phenomenon than a recipe to success.
We can move away from the specific ranking creation questions. I want to get some insight on S-tier tournaments which have been a hot topic the entire season. What shift caused us to go from 4 S-tier tournaments in v.3 to 10 in v.4?
That became a meme, too. The tournament tier system(TTS) is something we enacted in season 2. Before, people had trouble using the right terminology for events. People would flip major and premiers and regions with super regionals. It got really confusing and eventually we called them tiers (1-4), then moved to traditional tier list (S-C). That helps with marketing and investments when people describe their tournaments being an S tier or B tier. Season 3 was before the Summer of Smash, and the points were really aimed towards pot bonus and attendees. This led to Japan having a hard time to get ranked and the US had the easiest time due to player pools and talent available. Season 4 we did away with pot bonus and made it either top player density or attendance + pot bonus score decided the rank of the event. We wanted to give credit to events that had a ton of players, but only 1 or 2 PGR players. Through our testing, we saw that top player density determines the most accurate result for rankings, not a tournament with 2,000 randoms. The biggest reason is that we had the Summer of Smash. We had saga's fueling this very repeated high attendance from top players. We figured with the TTS, how much each ranked player should mean for ranking tournaments, because if Zero was there vs. a top 40 player, the difference needed to be shown. If you can get a ton of top players under one roof, it's for a reason. There was a saturation of events that involved all of them, all the time. They were, for all intents and purposes, S-tier tournaments. Many people saw S-tier as a ranking only to be given to the huge, once a year tournaments like EVO or Civil War. At one point we had Super Smash Con into Shine and that was proceed by SCR saga, which were all S-tier events. People were saying "S-tier this" or "S-tier that," but it was also the summer of smash. That fueled everyone's availability and desire to compete.
Next up, I want to talk about earning your rankings and how that goes down. This situation, you have two players that have wins on Captain Zack, one at the start of the season and one at the end. His ranking differs by about 10 spots, does this change how impactful the win is?
We have some top players that are sort of upset because there are some people that come out of nowhere and some that fall hard (reasons being work, school, character isn't working out like Ranai). Of course, Ranai was PGR'd the season before and he fell off. Lima and Mistake are examples of ones that come out and are instantly top 25. So people have been saying you beat ally at the beginning of the last season and he was just crowned top 10. Now, you know he has fallen off a great bit. So, his stock has plummeted in the sense of him moving outside of the top 5/10. Beating ally in the beginning vs. now, you would need an weekly updated system to validate that. Of course, if you beat him then, everyone knows you have a top 10 win, but at the end of the season he isn't regarded as the same. The constantly updated ranking would be able to handle that. However, we take in wins simultaneously at the end as people have done in the past 6 months. Because the problem validates Ally so highly, you don't do the same for Lima/Mistake. If you tell anyone a win on Lima/mistake doesn't matter, they will think you are crazy. If you take the whole season into account, you can work out all the kinks there. You would be asking for a different ranking system to handle this. We would need a lot more data to justify doing that. Being able to give everyone a passport of being top 50 helps them market themselves and that's important. I don't know if your next question is about live rankings, but if we are constantly changing and 30 people have claim to top 10 in a short span(due to the inconsistent nature of the game and lack of loads of data), it starts to mean less.
How much would my bribe have to be to get 42 on version 5?
For you to get on it? *Laughs* That's funny. We don't do bribes.
What led you to want to do a more cold, analytical ranking system instead of a ballet akin to the MIOM top 100 rankings? You guys do have an X-factor, but that isn't the official ranking.
That's a great question because ... It's funny, because there is a lot of melee interest in a PGR system because they feel the MIOM system is biased and it is also what Tafo thinks is the most appropriate. ELO/True Skill/Glicko (different type of ranking algorithms) all fall apart after top 20. Our system is a series of matrices and takes these simultaneous set counts and who you outplace, and that's only possible because of smash.gg and all the information available. Their system kind of just works for them and a decade worth of experience when it comes to this. It is a popularity contest but they take it seriously. What led me to have it be based just on statistics is that our scene and game is very new. Our results are always changing. If you don't go to one event you are washed and aren't relevant. If you come back and win, people then remember you and think you could be the best. Through our testing, we see players forget players wins/loses that they achieve in a specific season and believe stuff that happened in other season happened more recently. People obsess over peaks and upsets. Taking that into account and how young our scene (age and age of game), the amount of bias that would be seeping through a ballet system would not work out. We see it with our X-factor voting. People don't understand the numbers behind results because they will hold a high placement as very important even though maybe they did not beat anyone ranked in the top 10 to win the event, such as Dabuz at Civil War. This is why we did away with placement points for the rankings.
One quick question, do you ever think the rankings would expand to 100 players?
My personal opinion is no, and another point is feasibility and it's not appropriate. Melee has a deeper skill pool because of the age of the game. They are a very different community no matter how much people like to lump the games together. For the rankings and our testing, if you notice the score of everyone throughout the rankings, everyone starts to get really close. Everyone is compared to the score of #1 at 100. You start to see the number decrease and the gap in the higher levels. You go from 100 to 97, then 95. Sometimes the gap is only by .1. As you go lower, it compresses so much. The only reason we do Area 51 is because the difference between being 51 and 55 is so minuscule it almost doesn't matter. The top 25 is pretty compact, but after that everything can shift pretty far. It's like a sniper, if they shift 2 inches, miles away it is a mile long change. In our test of 2017, concurrent and unique entrants is around 5,000. By having a ranking of the top 50, represent about the top 1% and is very exclusive. It would start to tear away at the exclusivity and honestly, it would be so much work.
How do you feel about the people on reddit who make the TL;DW (Too Long; Didn't Watch) comments that make it so people don't want videos? We have seen Esam get into a debate about it diminishing the work he puts in, and I can even speak to the same thing as a content creator.
That's a whole different animal with digital media. We have seen a closure of a lot of sites. When it comes to content production, we live in a weird time where everyone wants everything immediately and quickly digestible. We see a lot of news publications go for ongoing videos and 5 to 10 words being flashed at the same time. You consume it in 2 minutes and you know all the information, but if it was an article it wouldn't be read. People are reading now more than ever, but they do not read for long. Of course you put this 10 minute video out and it might require all your faculties and they ask for the TLDW and people will not clicked on what you worked on. Twitter has the same problem. They have taken pictures of the final rankings and when that is posted, it will be more successful than the original tweet about the rankings. We had a problem where our video post did not do nearly as well as when CharlieDaKing posted his ranking. There is just a bigger audience than who is being hit by the video. In the past we did articles and narration and that's 20 or 30 pages of scripts. You had the problem of people who know the scene did not care about what was said because they knew. The people outside the scene do not know any comparison. Now you are talking to the middle and they already know the general info. We used to do a paragraph or two for each player, but people would gloss over that or not even visit the website so they could start debating the rankings on reddit. We kind of like adapted to the times and got rid of narration. Even if we had Morgan Freeman, you have audio and visual cues going off at once, it would be too much. This leads to good commentary and discussion.
People just want their content immediately. How do you monetize things that can be copy and pasted? Sites are going down, forums too because everyone is just talking on discord or twitter. We say to ourselves, if it can be a hit, we need to make it that. We made it something that anyone consume and didn't need to be translated. It's a huge multi-dimensional thing you have to grapple with the internet being the way it is, you always have to be thinking about how it can be shared if you aren't sharing it. It has to be clear and something that cannot be misunderstood so we try our best with all of that. We think we've had some good success so far.
Well, that raps up everything I had to ask. Thanks for coming on to talk and congratulations on making another successful version of the PGR, especially with an extremely clean set of videos for it. Please let the people know where they can find you online to keep up with you and your sponsors.
My Twitter is @PG_Suar and you can find the PGstats team on twitter @ThePGStats. We do a lot of live updates for tournaments on that account. I want to really shoutout @cooder_SSB4. His editing has been ridiculous and has blown this release out of the water. @PandaGlobalPG has been incredible as always and each season Allen, the CEO, and David, Co-founder, mentor me a lot through understanding how releases work and handling criticism. The partners of PG, Hyper-X and Geico, they have been awesome to help spread the message. It's been a fun time, and we get better every season.
Here is a link to everyone a part of the PGStats team. Consider giving each and everyone of them a follow.
Credits: Project Manager - suar @PG_suar
Writer - Dom @PG_dom
Analyst - Zan @PG_Zan
Analyst - Juddy @PG_juddy96
Rankings Consultant - PracticalTAS @PracticalTAS
Systems Engineer - Giant @PG_GiantFGC
Support Graphics - Spike @PG_Spike
Contributing Author - Mayday @PG_Mayday
Secondary Contributing Author - @PopiSSB
Project Mentor - SamuraiPanda @PG_SamuraiPanda
Videographer - Kud - @cooder_ssb4