SUMMARY/UPDATE at API fix from Reddit Unknown6 Team

[Mad4poko]

Just a new Tread to update you guys in news about work done for API fix, later on working at config for new bots. Post on, ask and deliver, all info is welcome.
Just sharing information post at Reddit pokemongo dev, I'm not part of the team or development, only non-programmer trying to help out in a working bot...and yes I'm against multi accounts and selling accounts...

[hazedoff]

I know a dev who's already fixed it, the fact the rest of you so called 'devs' are still scratching your heads at it is hilarious, none of you have a clue a part from Milas on how even the first API was originally created, never mind fixing the changes in the new one on your own, you just all hopped off his API and made your own little bots, probably even sold subscriptions to people, and are now frantically trying to work out whats changed, its a hilarious show to watch.

[Mad4poko]

SUMMARY/UPDATES
API stopped accepting requests from any sources which are not the actual client. The API needs a value "unknown 6", this value was already in the API in previous versions, but now the server is validating it. Only the actual client can create a valid "unknown6". We dont actually 100% know that it is indeed "unknown6" that is being validated, but it would make sense since its a big piece of data which isnt recreateable.

It is not as easy as locating where any updates made changes because the unknown6 was already being calculated and sent in previous versions but not validated by the server.
It doesnt really matter exactly what values go into the unknown6. Cracking/bruteforcing the code is impossible because the key alone wouldnt do it. We need to get to the piece of code that makes "unknown6". The key and the way to calculate unknown6 is somewhere within the code and were trying to find it.

We are trying to locate where the app calculates unknown6 in order to be able to recreate out own valid unknown6's. If we do that we have a working API again.
This is hard because parts of the code are not easily accessible. We need people that can decompile and document parts of the code!

GMT +1, 14:00 - Breakthrough? The programmers think they have found where unknown6 is created. Now the it still needs to be recreated and hope it actually works, that unknown6 really is what broke the code.

GMT +1, 14:30 - The dev discord has gone private due to people claiming the breakthrough as their own. They are still working doubletime on it! I am locked out on the discord too, so no more updates from me I guess. They let me in (16:20).
Unknown6 is indeed related to API changes, meaning our worst fear is not true. That would be when we would be able to recreate the unknown6, but that was not what broke the API. In that case everything we did would be worthless. We are on the right track.

GMT +1, 16:30 - The stuff being done is very technical. From my understanding we know where unknown6's core is created. From there we are able to see what inputs it takes and which functions it calls for further encryption. We are in a steady process of uncovering more steps of unknown6 it's creation. We've still got some steps to do..

GMT +1, 18:00 - Some important part of the encryption method has been decompiled, meaning we can now read it, and run the code through the decompilation when the other parts of the encryption have been found.

GMT +1, 19:30 - One step closer to fully determining the input.

GMT +1, 20:30 - Breaktrough #2: Two pieces of the unknown6 creation-code got linked together. We figured out where the encryption is called. As mentioned earlier we have the decompiled encryption.

GMT +1, 21:15 - We now need to do 2 things:
Get the decompiled encryption into a usable state. The encryption is a custom encryption and the decompiled file was over 200 pages long. People are working on it and it is not the hardest part but it has to be done. (slow but steady)
Figure out the last pieces of input, this could prove to be the difficult part. There is 3-4 fields remaining and every field that we figure out is a minor breakthrough.

GMT +1, 22:30 - No news, other than "they are working on it", but I thought I'd write something anyways, a reflection on the last 24 hours.
It has been facinating to see the devs from this sub work together to crack the unknown6. This is the same thing Ingress-hackers never defeated. But the POGO-dev community is bigger. I have seen people work on it 20 hours out of the 24 that the API-change is live. /u/keyphact hasnt slept for 40 (seriously go to sleep). These people are tirelessly, determined. I feel like we can do this.
We found the core creation place of unknown6 in mere hours. The encryptionfunctions were decompiled and the place where its called has been found. 10% of the input and the usability of the encryptionfunctions is whats left. Were so close, yet so far away. Will we solve this?

[Mad4poko]

GMT +1, 23:30 (sorry wrong timestamp previously) - We have much of the encryption understood. We however still dont know, how exactly the input is stored (protobuffer), this issue is very complicated. This is needed to track down the remaining inputfields.

GMT +1, 01:30 - We've got the encryption fully working (although we dont fully understand it)! You could call this breakthrough #3. The primary thing we are working on is getting the protobuffer.
This is a journey for me also. It is hard to keep up with what the devs are doing. What is a "protobuf format" for example? I am told it sits between the input and the encryption. It takes the inputvalues, rearranges them and sends them off for encryption. Like a blueprint for the inputdata.
Now we have the encryptionpart fully working, but we cannot backtrack to the input because we dont know how the blueprint arranged the inputvalues. Therefore we are making our own blueprint (protobuf-format)! Backtracking one step at a time. As we work on our protobuff format the input will become clear hopefully.

GMT +1, 03:30 - No major news. We're working on it and making progress.
I do want to make this another moment of reflection, the logistical nightmare of getting a community to work together like it has. It was a nightmare, without a clear solution, where the mods had no "good" choice.
It started off small: an open discord channel in which everyone could talk, working together to fix the API. It became clear this wasn't as easy as we thought.
Meanwhile the amount of people in the channel talking grew and grew. This however led to huge amounts of spam, most commonly "When is the API ready/What happened with the API". The situation became unworkable and we had to restrict talking rights on the discord.
This situation also became unworkable, people were claiming our progress as their own and they were also giving the community false hope as in how fast a new API would be made. On top of that the mods were still being spammed to death with requests for talking rights in the channel. We decided to hide the channel completely.
We tried a secondary channel, in which people could prove themselves worthy. But this channel started to get the same problems as the primary channel had initially. As well as people in the channel being understandibly angry at the mods, because they had no access to our primary channel they were doing the work we did hours ago.
Right now we are moving to transparancy again. We made the primary channel readable for everyone again. And hopefully noone will abuse this. We have also made an open-to-everyone github: https://github.com/pkmngodev/Unknown6 && https://github.com/pkmngodev/Unknown6/wiki
What can we learn from this though? I think there is no "solution" to this problem. Instead I want to thank the mods for putting in ridiculous amounts of work, merely to ease the pain of an unsolvable situation.

[Mad4poko]

5 August 2016, GMT +1, 13:00 - This is a cool tech-read on what we're doing right now: https://github.com/pkmngodev/Unknown...ment-237754201

GMT +1, 13:30 - No major news: right now it's a grind. We're working on the protobuf, we've renamed some more fields succesfully but there is still a big chunk of unknown left. We've also made progress on mapping all the functions that are called in the encryption, we are working to fully understand the encryption. Tracking the output back towards input is proving to be a tricky and tedious job.

I will also be answering some comments. Quick FAQ:
Q: I think I am qualified, how can I join to help?
A: I am sorry, but at the moment the primary channel is not open for applications. You can help however, we have a public repo where you can contribute and make a pull request: https://github.com/pkmngodev/Unknown6/wiki && https://github.com/pkmngodev/Unknown6.

Q: The devs should try x.
A: I have no idea what you're talking about, but I am sure the devs have thought of it. If you really think you have a brilliant discovery be sure to pass it on somewhere in the discord.

GMT +1, 15:30 - No major news again on the coding front, which was expected, its a grind.
I am updating to tell you that we've set up a reddit live thread: https://www.reddit.com/live/xdkgkncepvcq. The reddit livethread will contain more technical updates, expect to see terms you dont understand if you are not an experienced coder. If the devs don't update it they are busy coding. We've also set up a twitter, which will more accessible in terms of language. The twitter can be found at: https://twitter.com/pkmngodev, I will tweet whenever I update this comment (and they've given me access) They put me in charge of the Twitter.
We've also made the discord invite permanent, should not expire anymore, *fingers crossed*.
We want to keep you guys updated as well as not giving any room for fake twitter accounts.

5 August 2016, GMT +1, 16:00 - We have uncovered another field of the input! It feels good to have some progress finally. Don't get your hopes up YET, we still have another field to go, we are working to crack that too.

GMT +1, 17:00 - We have fully confirmed the earlier mentioned field of the input. Everyone is in a good mood, we're making progress.

GMT +1, 18:00 - We think the field we are trying to crack if connected to the field we just cracked. Hopefully that helps us.

GMT +1, 19:00 - We would like to repeat that the API-cracking community does not support bots. We are here to crack the API, thats it. That said we would like to confirm that Niantic can detect any MITM apps, these are apps that somehow modify data sent to the server. For example an app that ensures a perfect pokeballthrow. If you used an app like that Niantic could know.
We do not know whether you'll get banned for using such an app, we merely confirmed that Niantic could (theoretically) detect it. And it is not our concern, our concern is cracking the API.

[InvisibleTouch]

Detailed info for those who are interested:
The new struct, which we are calling Signature (previously Unknown6), authenticates every request made to the API. The structure of Signature turns out to be made up of a few more structures and values. We've managed to figure out what some of those are, leaving only a few remaining.

The integer unknown_10 is a hash generated by Pokémon Go. It is made by taking the user's current authentication ticket in protobuf format (app specific; different from their PTC or Google login token) and running it through a hashing function called xxHash32.
While our understanding of xxHash32 is not complete, from what we know, this hash function takes three values: the first being some data to hash, the second being a 32-bit integer to use as a "seed value" to start the function with, and the third being how many bytes from the data to use for hashing. In this case, the seed value has been determined to be 0x1B845328. This spits out a new 32-bit integer.

Now we take this integer and use it as the seed value to hash our current latitude, longitude, and altitude. The resulting 32-bit integer is unknown_10.

Similarly, the integer unknown_20 is another hash generated by the game. We take the current latitude, longitude, and altitude, as before, and hash it once again using xxHash32, with 0x1B845238 as the seed value. The resulting 32-bit integer is unknown_20.

We're not 100% sure about how to generate unknown_22, but it acts as a check to prevent users from simply copy-pasting API requests from the app. In other words, it prevents replay attacks. unknown_22 is time based and changes every few minutes.

Finally, unknown_24 (now known as request_hash) is an array of hashes, one for each function call being called via the API. (So, for example, getting the player info and inventory list in the same API call would result in two hashes.)
To generate request_hash, we start by hashing the authentication ticket with a seed value of 0x1B45328, but this time using the hashing function xxHash64. This becomes our new seed value for the remaining hashes.

Now, for each RPC function call included in the request, we take its protobuf data and hash using xxHash64 using the xxHash64 hash of the authentication ticket. The set of all these hashes is our request_hash.

[Mad4poko]

https://www.reddit.com/live/xdkgkncepvcq/

For update from LIVE Reddit thread for pokemongodev

[hazedoff]

lmao so weak

[valebiets]

SUMMARY/UPDATES
API stopped accepting requests from any sources which are not the actual client. The API needs a value "unknown 6", this value was already in the API in previous versions, but now the server is validating it. Only the actual client can create a valid "unknown6". We dont actually 100% know that it is indeed "unknown6" that is being validated, but it would make sense since its a big piece of data which isnt recreateable.
It is not as easy as locating where any updates made changes because the unknown6 was already being calculated and sent in previous versions but not validated by the server.
It doesnt really matter exactly what values go into the unknown6. Cracking/bruteforcing the code is impossible because the key alone wouldnt do it. We need to get to the piece of code that makes "unknown6". The key and the way to calculate unknown6 is somewhere within the code and were trying to find it.
We are trying to locate where the app calculates unknown6 in order to be able to recreate out own valid unknown6's. If we do that we have a working API again.
This is hard because parts of the code are not easily accessible. We need people that can decompile and document parts of the code!

GMT +1, 14:00 - Breakthrough? The programmers think they have found where unknown6 is created. Now the it still needs to be recreated and hope it actually works, that unknown6 really is what broke the code.

GMT +1, 14:30 - The dev discord has gone private due to people claiming the breakthrough as their own. They are still working doubletime on it! I am locked out on the discord too, so no more updates from me I guess. They let me in (16:20).

Unknown6 is indeed related to API changes, meaning our worst fear is not true. That would be when we would be able to recreate the unknown6, but that was not what broke the API. In that case everything we did would be worthless. We are on the right track.

GMT +1, 16:30 - The stuff being done is very technical. From my understanding we know where unknown6's core is created. From there we are able to see what inputs it takes and which functions it calls for further encryption. We are in a steady process of uncovering more steps of unknown6 it's creation. We've still got some steps to do..

GMT +1, 18:00 - Some important part of the encryption method has been decompiled, meaning we can now read it, and run the code through the decompilation when the other parts of the encryption have been found.

GMT +1, 19:30 - One step closer to fully determining the input.

GMT +1, 20:30 - Breaktrough #2: Two pieces of the unknown6 creation-code got linked together. We figured out where the encryption is called. As mentioned earlier we have the decompiled encryption.

GMT +1, 21:15 - We now need to do 2 things:

Get the decompiled encryption into a usable state. The encryption is a custom encryption and the decompiled file was over 200 pages long. People are working on it and it is not the hardest part but it has to be done. (slow but steady)
Figure out the last pieces of input, this could prove to be the difficult part. There is 3-4 fields remaining and every field that we figure out is a minor breakthrough.

GMT +1, 22:30 - No news, other than "they are working on it", but I thought I'd write something anyways, a reflection on the last 24 hours.
It has been facinating to see the devs from this sub work together to crack the unknown6. This is the same thing Ingress-hackers never defeated. But the POGO-dev community is bigger. I have seen people work on it 20 hours out of the 24 that the API-change is live. /u/keyphact hasnt slept for 40 (seriously go to sleep). These people are tirelessly, determined. I feel like we can do this.

We found the core creation place of unknown6 in mere hours. The encryptionfunctions were decompiled and the place where its called has been found. 10% of the input and the usability of the encryptionfunctions is whats left. Were so close, yet so far away. Will we solve this?

if you can edited like that, that would be good and more easy to understand.. thank you.

[wwe2030]

i guess we will have a new bot soon right

[Miuobyshi]

lmao so weak

you are ignored over and over and over, why so mad ?

[ZeroDayGhost]

you are ignored over and over and over, why so mad ?

Probably because he's weak himself and the only way to ease it is to project onto others.

[rsxtypers]

Just a new Tread to update you guys in news about work done for API fix, later on working at config for new bots. Post on, ask and deliver, all info is welcome.
Just sharing information post at Reddit pokemongo dev, I'm not part of the team or development, only non-programmer trying to help out in a working bot...and yes I'm against multi accounts and selling accounts...

I find it funny that you guys cracking the api say you so not support bots...if that is true you would not crack it at all. So please stop with the moral highground. Other then that keep up the great work

[mpslayer]

I know a dev who's already fixed it, the fact the rest of you so called 'devs' are still scratching your heads at it is hilarious, none of you have a clue a part from Milas on how even the first API was originally created, never mind fixing the changes in the new one on your own, you just all hopped off his API and made your own little bots, probably even sold subscriptions to people, and are now frantically trying to work out whats changed, its a hilarious show to watch.

Uhh didn't Ferox release first?

[chooseusername]

I know a dev who's already fixed it, the fact the rest of you so called 'devs' are still scratching your heads at it is hilarious, none of you have a clue a part from Milas on how even the first API was originally created, never mind fixing the changes in the new one on your own, you just all hopped off his API and made your own little bots, probably even sold subscriptions to people, and are now frantically trying to work out whats changed, its a hilarious show to watch.

Well who is this dev :~) ?