status-security/incidents/2020/2020-02-07_v1-app-prelaunch...

---
title: V1 app store pre-launch post-mortem
tags: postmortem, security, transparency
---

# Publishing Status V1 post-mortem - Draft v0.1

## Date
2020-01-07

## Author
Corey Petty

## Summary

The Status app was published to v1.0.0 on the Android Play Store before it was intended. 

## The Impact
At the point of writing this, there were 

This affects users in a multitude of ways, and violates the Status principles in the following ways:
- **principle 1** - 

### Scope

## Trigger

## Resolution
- Removal of the application from production in both iOS and Android Play app stores.

## Action Items
| Action Item | Type | Owner | Reference | Notes |
| ----------- | ---- | ----- | --------- | ----- |
| Development of Release Process | prevent | ??? | **TODO** | description here |
| Remove publication from app stores | mitigate | Andre | **DONE** | descripton here |
| Figure out auto-update process | prevent | jakubgs | **DONE** | see [Jakub's recovery process](#jakub-building-auto-update-recovery) | 
| ... another item | mitigate/prevent | ?? | **TODO** | descripton here |
| explore future beta channel options | improvement | andre | **TODO** | descripton here |

## Lessons Learned
### What went well
- It appears as though not many users were impacted negatively in the time the version was in the app store and upgraded/installed
- 

### What went wrong
- 500+ users were upgraded from beta to V1 potentially without knowledge of potential breakage and time to back up appropriate material
- There was a lack of coordination on the release process.

### Where we got lucky
- we have a relatively low userbase

## Timeline

( All times EDT )

| Date-Time  | Description |
| ---------- | ----------- |
| 2020-02-05 13:21 | Apple submission set to `In Review` |
| 2020-02-05 17:21 | Apple submission set to `Pending Developer Release` |
| 2020-02-06 07:29 | @andre announces on Discord that app was approved and is live |
| 2020-02-06 07:38 | Android set to `Ready for Sale` by `me@andremedeiros.info` |
| 2020-02-06 07:46 | @jakubgs asks in Discord if Android auto-updates and is quickly confirmed that it does by @cammellos |
| 2020-02-06 07:7:47 | @cammellos says in Discord that `that probably will break a few apps` |
| 2020-02-06 07:7:56 | @andre confirms on Discord that apple app store listing is live |
| 2020-02-06 07:57 | @petty states in Discord that `we should warn users as best we can to make sure not to upgrade before they know they have their keys` |
| 2020-02-06 08:10 | @rachel asks on Discord if it is possible for an automatic update on Android from beta to public version |
| 2020-02-06 08:22 | @andre states in Discord that `To make changes to the app name, category or privacy policy, create a new app version. All other changes will be immediately available on the App Store.` |
| 2020-02-06 08:23 | @andre suggests in Discord to push a v1.0.1 with small fixes discussed in previous hour or so based on new discoveries in Android App Store policies |
| 2020-02-06 08:35 | @rachel notices on Discord that the app name still has `beta` in the title, @jakubgs confirms and looks up, gives info: `"Same. There's no re-uploading - you just change the state of the release." "Important INFO about ROLL-OUT TIME. If you published your app to beta-channel breforehand and then click "RELEASE TO PRODUCTION" button, do not expect it will be roll out immediately, it will still take up to 24 HOURS to be live. "`|
| 2020-02-06 08:45 | Discord chatter discovers that being int he beta program shows `beta` in the title |
| 2020-02-06 08:54 | after a few requests, @andre in Discord says that he has removed the old YouTube video but it takes a while to update the page |
| 2020-02-06 08:58 | @jakubgs in Discord asks for people to take a look at status.im site changes to reflect V1 |
| 2020-02-06 08:58 | @jakubgs asks @andre in Discord if we should start rolling out nightlies again, as they are currently pointing to beta binaries |
| 2020-02-06 09:01 | @arnetheduck says in Discord that he noticed status was published, and asked what would happen if he upgrades from current beta install, @jakubgs informs him he needs to backup seed phrase, uninstall, and install V1, @arnetheduck then asks people to assume he's just a user that has auto-update enabled in play store |
| 2020-02-06 09:10 | @arnetheduck says in Discord: `@andre er, well, the app was available on the app store before - presumably some users installed it - normies have auto-update installed, and now I guess they will be faced with.. potential explosions? I don't know how the versioning is handled on release. if I was doing a completely incompatible release that doesn't include migration code, I'd consider releasing it as a new application and renaming the old application to "obsolete" or some such`, @cammellos responds with `I thought that was the plan, or that the app would be seen as a completely new one once we leave the "beta" program`, @andre confirms that that was his assumption as well |
| 2020-02-06 10:11 | @andre asks in Discord to avoid mentioning release until comms are published |
| 2020-02-06 10:31 | @rachel announces in Discord: `buttt if they've been auto updated they'll have to recover + if seed phrase is lost, it's lost.  post is live: https://our.status.im/announcing-status-version-1-0/` |
| 2020-02-06 10:36 | @andre finds that existing beta users can be affected by auto-update, @rachel mentions that this can lead to loss of funds potentially, and that she thinks we need damage control comms for usres who are forcibly upgraded |
| 2020-02-06 10:43 | @andre announces he's making the app unavailble in all countries for now, which should stop rolling update |
| 2020-02-06 11:03 | Android set to `Developer Removed From Sale` by `me@andremedeiros.info` |
| 2020-02-06 11:05 | @fizzgig says `It's unfortunate, I'm sorry to point this out because everyone has been working so hard, the app looks fantastic and feels solid, but this should be a valuable lesson to those who argued against doing a migration strategy. I hope it's clear we can't have these things happen moving forward. Accidents happen, and thank you everyone for your response.` |

## Other Mitigation Actions

### Jakub building auto-update recovery
conversation taken from `core-infra` of discord:
```
jakubgsToday at 8:57 AM
well, there's no PR
I just rebuilt and old APK
for 0.13.1
with new build number
so someone can get that APK
and install it on top of 1.0.0
and recover their old account
which used a different db schema/file
http://status-im.ams3.digitaloceanspaces.com/StatusIm-200211-140000-7f38c5-release.apk
this is the APK
because it's versionCode is higher than that of v1.0.0 we accidentally released a user that has lost access to their old account can recover it installing that APK over the v1.0.0
```
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00			`---`
			`title: V1 app store pre-launch post-mortem`
			`tags: postmortem, security, transparency`
			`---`

			`# Publishing Status V1 post-mortem - Draft v0.1`

			`## Date`
			`2020-01-07`

			`## Author`
			`Corey Petty`

			`## Summary`

updates to v1 PM 2020-02-11 14:06:05 +00:00			`The Status app was published to v1.0.0 on the Android Play Store before it was intended.`
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00
			`## The Impact`
updates to v1 PM 2020-02-11 14:06:05 +00:00			`At the point of writing this, there were`
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00
			`This affects users in a multitude of ways, and violates the Status principles in the following ways:`
			`- principle 1 -`

			`### Scope`

			`## Trigger`

			`## Resolution`
updates to v1 PM 2020-02-11 14:06:05 +00:00			`- Removal of the application from production in both iOS and Android Play app stores.`
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00
			`## Action Items`
updates to v1 PM 2020-02-11 14:06:05 +00:00			`\| Action Item \| Type \| Owner \| Reference \| Notes \|`
			`\| ----------- \| ---- \| ----- \| --------- \| ----- \|`
			`\| Development of Release Process \| prevent \| ??? \| TODO \| description here \|`
			`\| Remove publication from app stores \| mitigate \| Andre \| DONE \| descripton here \|`
			`\| Figure out auto-update process \| prevent \| jakubgs \| DONE \| see [Jakub's recovery process](#jakub-building-auto-update-recovery) \|`
			`\| ... another item \| mitigate/prevent \| ?? \| TODO \| descripton here \|`
added V1 FAQ 2020-02-14 15:33:54 +00:00			`\| explore future beta channel options \| improvement \| andre \| TODO \| descripton here \|`
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00
			`## Lessons Learned`
			`### What went well`
added V1 FAQ 2020-02-14 15:33:54 +00:00			`- It appears as though not many users were impacted negatively in the time the version was in the app store and upgraded/installed`
updates to v1 PM 2020-02-11 14:06:05 +00:00			`-`
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00
			`### What went wrong`
updates to v1 PM 2020-02-11 14:06:05 +00:00			`- 500+ users were upgraded from beta to V1 potentially without knowledge of potential breakage and time to back up appropriate material`
			`- There was a lack of coordination on the release process.`
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00
			`### Where we got lucky`
updates to v1 PM 2020-02-11 14:06:05 +00:00			`- we have a relatively low userbase`
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00
			`## Timeline`

			`( All times EDT )`

			`\| Date-Time \| Description \|`
			`\| ---------- \| ----------- \|`
			\| 2020-02-05 13:21 \| Apple submission set to `In Review` \|
			\| 2020-02-05 17:21 \| Apple submission set to `Pending Developer Release` \|
			`\| 2020-02-06 07:29 \| @andre announces on Discord that app was approved and is live \|`
time changes for post mortem 2020-02-07 23:06:23 +00:00			\| 2020-02-06 07:38 \| Android set to `Ready for Sale` by `me@andremedeiros.info` \|
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00			`\| 2020-02-06 07:46 \| @jakubgs asks in Discord if Android auto-updates and is quickly confirmed that it does by @cammellos \|`
			\| 2020-02-06 07:7:47 \| @cammellos says in Discord that `that probably will break a few apps` \|
			`\| 2020-02-06 07:7:56 \| @andre confirms on Discord that apple app store listing is live \|`
			\| 2020-02-06 07:57 \| @petty states in Discord that `we should warn users as best we can to make sure not to upgrade before they know they have their keys` \|
			`\| 2020-02-06 08:10 \| @rachel asks on Discord if it is possible for an automatic update on Android from beta to public version \|`
			\| 2020-02-06 08:22 \| @andre states in Discord that `To make changes to the app name, category or privacy policy, create a new app version. All other changes will be immediately available on the App Store.` \|
			`\| 2020-02-06 08:23 \| @andre suggests in Discord to push a v1.0.1 with small fixes discussed in previous hour or so based on new discoveries in Android App Store policies \|`
			\| 2020-02-06 08:35 \| @rachel notices on Discord that the app name still has `beta` in the title, @jakubgs confirms and looks up, gives info: `"Same. There's no re-uploading - you just change the state of the release." "Important INFO about ROLL-OUT TIME. If you published your app to beta-channel breforehand and then click "RELEASE TO PRODUCTION" button, do not expect it will be roll out immediately, it will still take up to 24 HOURS to be live. "`\|
			\| 2020-02-06 08:45 \| Discord chatter discovers that being int he beta program shows `beta` in the title \|
			`\| 2020-02-06 08:54 \| after a few requests, @andre in Discord says that he has removed the old YouTube video but it takes a while to update the page \|`
			`\| 2020-02-06 08:58 \| @jakubgs in Discord asks for people to take a look at status.im site changes to reflect V1 \|`
			`\| 2020-02-06 08:58 \| @jakubgs asks @andre in Discord if we should start rolling out nightlies again, as they are currently pointing to beta binaries \|`
			`\| 2020-02-06 09:01 \| @arnetheduck says in Discord that he noticed status was published, and asked what would happen if he upgrades from current beta install, @jakubgs informs him he needs to backup seed phrase, uninstall, and install V1, @arnetheduck then asks people to assume he's just a user that has auto-update enabled in play store \|`
			\| 2020-02-06 09:10 \| @arnetheduck says in Discord: `@andre er, well, the app was available on the app store before - presumably some users installed it - normies have auto-update installed, and now I guess they will be faced with.. potential explosions? I don't know how the versioning is handled on release. if I was doing a completely incompatible release that doesn't include migration code, I'd consider releasing it as a new application and renaming the old application to "obsolete" or some such`, @cammellos responds with `I thought that was the plan, or that the app would be seen as a completely new one once we leave the "beta" program`, @andre confirms that that was his assumption as well \|
			`\| 2020-02-06 10:11 \| @andre asks in Discord to avoid mentioning release until comms are published \|`
			\| 2020-02-06 10:31 \| @rachel announces in Discord: `buttt if they've been auto updated they'll have to recover + if seed phrase is lost, it's lost. post is live: https://our.status.im/announcing-status-version-1-0/` \|
			`\| 2020-02-06 10:36 \| @andre finds that existing beta users can be affected by auto-update, @rachel mentions that this can lead to loss of funds potentially, and that she thinks we need damage control comms for usres who are forcibly upgraded \|`
			`\| 2020-02-06 10:43 \| @andre announces he's making the app unavailble in all countries for now, which should stop rolling update \|`
time changes for post mortem 2020-02-07 23:06:23 +00:00			\| 2020-02-06 11:03 \| Android set to `Developer Removed From Sale` by `me@andremedeiros.info` \|
initial-v1-launch-postmortem 2020-02-07 23:02:06 +00:00			\| 2020-02-06 11:05 \| @fizzgig says `It's unfortunate, I'm sorry to point this out because everyone has been working so hard, the app looks fantastic and feels solid, but this should be a valuable lesson to those who argued against doing a migration strategy. I hope it's clear we can't have these things happen moving forward. Accidents happen, and thank you everyone for your response.` \|

updates to v1 PM 2020-02-11 14:06:05 +00:00			`## Other Mitigation Actions`

			`### Jakub building auto-update recovery`
			conversation taken from `core-infra` of discord:
			```
			`jakubgsToday at 8:57 AM`
			`well, there's no PR`
			`I just rebuilt and old APK`
			`for 0.13.1`
			`with new build number`
			`so someone can get that APK`
			`and install it on top of 1.0.0`
			`and recover their old account`
			`which used a different db schema/file`
			`http://status-im.ams3.digitaloceanspaces.com/StatusIm-200211-140000-7f38c5-release.apk`
			`this is the APK`
			`because it's versionCode is higher than that of v1.0.0 we accidentally released a user that has lost access to their old account can recover it installing that APK over the v1.0.0`
			```