Lessons Learned from building a Pokemon Go Raid Tracker

For the last 4 months I’ve spent my spare time creating a raid tracker for Pokemon Go.

In the game raids are temporary events at gyms (special entities in particular locations) that see one or more players collaborate on defeating a Pokemon that is given statistics much higher than they would normally have with the goal of then catching the normalised version of that Pokemon.

In August 2017 Niantic (the company behind Pokemon Go) announced a new type of raid, the Exclusive Raid (Ex-Raid) that saw players take on Mewtwo, the first mythic tier Pokemon. These Ex-Raids were by invitation only and had certain requirements to get the invitation (these are called Raid Passes in the game).

As an avid Pokemon Go player I set out to create a tool to help my fellow local players in Leeds, UK get one of these Ex-Raid passes.

Requirements

The requirements for getting an Ex-Raid pass at the time (it’s since changed) were that players would have needed to complete a raid at the gym ‘recently’.

The requirements at the beginning were very vague, as Niantic wouldn’t create an actual Ex-Raid in the game outside of the special event in August — until a month later when there were multiple waves of Ex-Raid passes after being experimented, where they changed the algorithm used to determine when an Ex-Raid would happen.

My initial solution

Given the requirements I was working with I sat down and started thinking about how I and others would be able to tell if they were on the right track to meeting the Ex-Raid criteria.

I decided that the best approach would be to ensure that players have done a raid at the broadest range of gyms possible and when they risked falling out of the ‘recent’ criteria (at the time believed to be 4–7 weeks) they would be prompted to raid at that gym again.

I also decided as the biggest concentration of gyms is in Leeds City Centre I would focus my initial set of gyms to those lying within the Leeds City ring road. This meant I could provide a coverage percentage to players and as I was manually entering the gym data I wouldn’t have to add all 600+ gyms in Leeds.

My evidence for this was:

• People were already tracking the gyms they raided at in spreadsheets
• The players I interacted with would unless they were very dedicated only go to raids within 3km of their home or work
• Each player gets a free raid pass a day

Fleshing out my idea

After coming to the conclusions I came to, I set about to validate my initial idea with the players that I regularly raided with. These players range from hardcore players who spend 4 hours+ a day playing the game, to those who just play on their lunch break.

For the most part people said that having some kind of app to track their raids on would be beneficial so they don’t miss out. The hardcode players were very interested in adding meta data to the raids so they could track if they caught the Pokemon or not (there was a rumour that this affected the EX-Raid invite).

After this initial conversation I sat down with one of my friends and ran a bit of a co-design session. He runs the local Facebook group for players in Leeds (a 3,000 strong community) as well as the discord and was working on his own solutions to other problems the community faced.

Out of our sessions I was able to create a set of personas covering:

• Hardcore players (like understanding their performance, hate underperforming casual players)
• Casual players (like tools that give them advantages so can perform at similar level to hardcore players, hate investing time)

I also created a user journey through the app which involved:

• Discover the app (direct, Facebook, Discord)
• Sign up / login
• Find gym that they want to log raid at
• Log raid
• Check / be reminded of progress

The core loop being the find gym, log raid, check progress parts of the user journey.

With the co-design sessions, personas and user journeys fleshed out I was then able to start building up a backlog of features, some of these were:

• Add live raid data to the app
• Allow players to add meta data to the raids
• Show players their raiding history and allow them to share infographics
• Give players an interactive tool to explore the raids that happened in Leeds so they can spot trends
• Add functionality to allow players to organise raiding parties (there’s a big issue with players arriving to find everyones already completed the raid)
• Allow players to track additional goals (raids count towards completing a number of goals such as players increasing level, players getting a badge for number of raids completed and levelling up the badge associated with the gym)

Validating my idea

After receiving positive feedback from my immediate player base I sat down in front of Sketch and started creating mock-ups based on my initial designs drawn up during the sessions I held with my friend.

The mock-up illustrated the user journey of a player logging in, finding a gym, adding a raid and seeing the progress gained after doing so.

I then posted the mock-ups to Facebook as well as asking people I’d not spoken to before at raids to give me feedback.

Aside from one person telling me ‘not to overthink it’ the general opinion was that the tool was useful, so I took this as a sign that I was heading in the right direction and started developing the product.

I set up a FeatureMap.co story map for the tracker and created a development project on the Github repo to store the backlog of development tasks ready for building the MVP.

Developing the MVP

I’m more at home with Python so I decided in order to deliver the MVP quickly I’d use Django, as it offers a lot out of the box and I’ve got experience with it.

For hosting I used Heroku as the free tier was more than enough for me and I love the integration it has with Github for deploying on pushes to a particular branch after the tests have passed.

I used Travis CI and Codacy for my continuous integration and code quality tools and set these up to run checks on every pull request I opened on Github. I like to work using feature branches, so these checks reduced the risks of bugs creeping in.

It took me a few nights (and a few mornings!) to get the MVP ready. Once I had it built and deployed I gave the link to the moderators of the Facebook page so they could start using it and raise any issues they found with it.

During this time there were about 100 raids recorded and no issues raised, so after prompting for feedback and receiving nothing negative I decided to launch the app on the Facebook group shortly after.

On launch I received one comment that pointed out something I’d stupidly left out, I forgot to add a means to search for a gym by name. I got this implemented the next night which delighted them, as they were expecting the comment to go ignored.

Getting live raid data

There are a number of maps available to Pokemon Go players, as the game is based on being in the right location at the right time. The players use these maps to plan where they will go in order to get Pokemon, attend raids or attack/defend gyms.

The two main maps that people use in Leeds are Pokehuntr and GoMaps. PokeHuntr has a sister map call GymHuntr for mapping gyms, where as GoMaps does it all on one map, but the map is limited to a particular region.

In order to get live raid data into the tracker app I needed to identify a data source. I could have used a personal instance of RocketMap but this would require maintaining accounts, as well as an additional server.

The Gymhuntr map is the biggest of the two, having no regional limitation and a more visible community; so I joined their discord and attempted to get in touch with the developers to see if they had an API I could use. After much waiting and pestering I finally got a reply about an API they have.

The GymHuntr API requires the knowledge of the Gym’s API key and will return only the team that currently controls the gym. You’re limited to 20 calls per IP an hour so this was not very useful for my needs and I moved onto GoMaps.

GoMaps had no public API, but a bit of reverse engineering allowed me to scrape the data, so I pulled together a proposal to the person who runs it. I explained the aim of my app, the number of calls they could expect and how I’d use the data.

They were hesitant at first as they worried about loss of income, as API calls wouldn’t load the advertising they use to fund the maps; however, I made a number of donations that helped to resolve some of the issues and they agreed to let me scrape the data from their site.

When I tried to run my scraping script from the production server, I ran into an issue: GoMaps uses Cloudflare to prevent bots accessing the server, so I was being hit with 503 errors when trying to get the data. Luckily there’s a really good Python library called cfscrape that manages the challenge presented to browsers and lets the requests go through.

Adding raids to the tracker

Before touching any code (aside from proving it was technically feasible to get the raid data) I went back to my sketchpad and drew up designs on how I’d integrate the raids into the UI.

The changes to the UI involved:

• Adding a banner to the gym to show there was a raid active
• Creating a new list on the gym page for active raids
• Showing the name of the Pokemon in the data entry screen where the user added the data for the raid

I then turned to the Facebook group again for feedback on a set of mock ups of the UI changes and made a few tweaks based on suggestions from people.

Once I was happy with the work that needed to be done I set out to implement these into the system.

The only hiccup I faced here was updating my BDD test suite to work with the new changes, as I had written them in such a way that I was abusing the context object that Behave (the test runner I used) passed between steps.

Once I had finished the implementation I launched the raids into the app and got a lot of really good positive feedback from the player base. One player even said that they stopped checking the different map websites, as having a list to focus on made it easier to plan their raiding schedule.

Another piece of feedback was that when retroactively adding raids (some players added their raids at the end of a raiding session) it would be good to have a list of the recent raids on the gym, so you didn’t have to remember what time you went to the raid. Similar to the search bar before, they were amazed at the turn around time and really happy to see their suggestion make it into the app.

Unfortunately adding a 3rd party source for real time data meant that I was dependent on the provider communicating changes to the data returned, which resulted in a number of ‘raid outages’.

To help me mitigate the surprise of the API changes, I set up Sentry on my Heroku stack. Sentry parses the application logs and acts as a dashboard to triage issues that happen in production. With this I could see when the scraping script failed due to an API change and could make a fix quickly.

Then one day GoMaps moved their API to contain a token that was refreshed with every call and I could no longer get data. Talking to them, it turned out the software they ran implemented this and they were setting up a public API.

Getting live raid data back

The public API for GoMaps isn’t the same as the one that is served to the maps’ frontend.

Instead it requires an API key that limits which data sets can be queried. The datasets returned are modelled in a manner similar to a relational database, so the raid data set has an ID to the gym instead of the gym name.

The documentation of the API also left me a bit underwhelmed, I was hoping for a ReSTful API with clearly defined structures for the endpoints but instead there’s one endpoint called /api and the collection / resource requests are passed as POST data.

At work I’d attended a talk about APIs and been introduced to Swaggerhub, a great website for defining API specifications; so in a bid to try and get the API improved for my use, I started to create a specification for the existing API and a more ReSTful version to illustrate how returning more than just a database table would help build a community around the GoMaps data set.

After initial communication I was able to get an API key and the SwaggerHub spec was well received, but that’s as far as it went. Development on the API wasn’t a priority and it was technically usable if you could parse it (it wasn’t returning a valid JSON document).

As a month or so had passed (I ended up putting the tracker development on hiatus having to prepare for a holiday in Japan) I decided that this API was the best I was ever going to get, bit the bullet and invested time in using it in it’s current state.

In mid December I was finally able to get raid data back into the app and this time I was getting data for the whole of Leeds, which meant I had no excuse not to expand the range supported by the tracker.

Expanding to cover all of Leeds

Now I was getting the raids for all the gyms in Leeds it made sense to have all those gyms in the tracker. The question was though, how to add 600+ gyms to a UI that is just a list of gyms.

I’d spent some effort before the development hiatus working on the design of this and had gotten feedback on some interactive mock ups I’d let people play with.

The general consensus was that any gyms they add raids to should be added to a list of gyms they want to track, as well as the ability to add and remove gyms from that list. Once they had that list populated, they wanted to see raids for the gyms in that list.

To achieve this I added a new page that would list the active raids, added a gym management page where users could search for a gym by name and add it to the list of ‘tracked gyms’ and unified the gym visit list so it was ordered to have the gyms players needed to visit at the top.

In order to get the gyms I resorted to copying the JSON response of GoMaps from the network tab of Chrome development tools and processing this to add the Gyms to the system. This worked, as the gyms wouldn’t update very often and it was quicker than negotiating the API I had having it’s permissions changed.

I launched the expanded version of the tracker to very little fanfare. During the time since the last feature on the tracker, the criteria for Ex-Raids had changed.

It was only gyms in parks that were getting the invites and these were linked to S2 cells (a way of segmenting maps at different zoom levels) so people no longer needed to try and achieve blanket coverage of gyms.

Detecting gyms in parks

In December 2017 Pokemon Go changed the data source that it pulled it’s map tiles from, ditching Google Maps for Open Street Map (OSM).

With the change of map provider came advantages to players, namely the Overpass API that Open Street Map has.

As the Ex-Raid criteria now specified gyms in parks, anyone could write a query, go to Overpass Turbo and get a list of features on the map that are considered to be parks.

I had spent a little too much time building the tracker app, refactoring and improving the infrastructure and process around the development to follow any of the information coming out about this so I was a little late playing around with the API.

To get something out there I wrote a very basic query to return the ways around the gyms that I had in the system. To do this I converted the location coordinates that I had to the gym in app - into a tile number at zoom level 19, I then created the bounding box by padding this tile number by 1 and running the query.

It wasn’t amazingly accurate but it was good enough to detect gyms that had Ex-Raids previously, so I shipped it. On doing so I got one bit of feedback which was to show not only gyms in parks but those that had Ex-Raids previously, so I added this too in the hope it was useful.

It wasn’t until Trainer Tips (a well known and respected Pokemon Go player and Youtuber) made a video explaining how S2 cells and Ex-Raid passes are linked that I actually found a good way of determining which gyms were viable Ex-Raid candidates.

Thanks to TheSilphRoad, a community of players who research Pokemon Go and try and reverse engineer how Niantic run the game, a proven way of predicting Ex-Raids as well as a number of other game mechanics were documented.

Based on research from players across the world they determined that Ex-Raids were linked to level 12 S2 cells with only one gym in an S2 cell hosting an Ex-Raid at one time.

To improve my way of detecting which gyms were in parks I looked into S2 cells and found the following tools:

• OSMcoverer a Go library to convert location coordinates into S2 cells and when supplied with a geojson document can determine overlaps of features and markers (i.e. if a gym marker is in a park feature)
• osmtogeojson a Node JS script to convert OSM’s JSON output to geojson
• geojson.io a website for rendering GeoJSON data over Open Street Maps and other map providers

I then combined them to:

• Get all the level 12 S2 cells for Leeds
• Query Open Street Map for features using the S2 cell as the bounding box
• Convert the data from OSM to GeoJSON
• Using the GeoJSON for the features, get a list of gyms that overlap with the features from OSM

As I was in good standing with GoMaps I decided to build a new repo with a Makefile to automate the scripts so they could use it on their website. They have since added a flag to the site to indicate if a gym is in a park.

Shutting down the tracker

After doing the work to add gym management, get the raids back on the site and add the indicators to show if the gym is viable for an Ex-Raid, I looked at the Google Analytics to see the number of active users.

Much to my dismay for the last month I’d not had any increase in users. Even though I was adding features that I was being told were valuable, no one was using them.

After a bit of a root cause amongst some friends I’ve decided to shut the tracker down and focus on some more fun problems in relation to Pokemon Go without being tied down to an existing platform that isn’t fit for purpose.

From the root cause we came to the following conclusions:

• The initial idea while meeting the vague specification at the time wasn’t applicable anymore
• Improvements on the maps people used meant there was no need to go to the tracker unless you cared about tracking your raids
• Technical issues with the raid data meant that people lost interest as the tracker ‘seemed to no longer work’
• Other members of the Facebook group had done their own S2 cell analysis before it was implemented into the tracker so it was old news by the time it was added
• It took too long to expand the tracker to support all of Leeds instead of the city centre

What building the tracker taught me

Now I’ve shut down the tracker and written this blog post about the development journey I’ve got a few lessons that I’ve learned to document.

• Building solutions to problems that haven’t been clearly defined will very likely lead you to focusing on the wrong area
• Focusing on the wrong area will make it harder to continue to deliver value to users when they realise your solution is wrong
• Users love it when their feedback is taken on board and they can see the influence they had in the development of the product
• Depending on 3rd party data sources requires communication from both parties, so be prepared to put in a lot of effort to work around changes made with no communication
• I discovered that I’m happier during the ideation, validation and MVP stages of a project than later on when the project is established and the changes are smaller

As this was my first personal ‘product’ I’d built it’s definitely helped me to understand where I want to be in the product life cycle. I’ve been a developer for the last 7 years but I now know that my heart lies at the stages before writing code and I’ll be looking to move into the discovery / UX field in my career.