Google’s code transparency feature alleviates some concerns, but does it go far enough?
Last November, Google announced that developers will be required to publish new apps on the Play Store using the Android App Bundle (AAB) format instead of an APK. Just the other day, Google reminded developers of this upcoming requirement, setting off a firestorm of controversy from users who believe that Google is killing APKs, eliminating sideloading, hindering third-party app stores, and whatnot.
It’s true that Android App Bundles are a pretty big departure from the classic APK format you might be used to, both as a user and as a developer. While there are quite a few benefits to using App Bundles, there’s one key aspect to making them that has some developers and security experts rightly concerned.
In this article, we’ll cover the criticisms we’ve seen of the switch to Android App Bundles as well as some proposed solutions, and we’ll also talk about Google’s proposed solution to these problems.
Before that happens though, we need to talk a bit about how app distribution works on Android in general. If you already know how app signing and App Bundles work, you can skip this part.
For the most part, apps on Android are distributed inside of APK files. An APK contains all of an app’s code and resources, along with some security features like a signing manifest. When an APK is installed, it’s basically just copied to a specific folder and added to an internal database of installed apps.
During installation, that app’s signature is also verified to make sure it’s valid. If the app is already installed, Android checks the new app’s signature against the one that’s already installed. If the signature isn’t valid or doesn’t match, Android will refuse to install the app.
That signature checking is an important part of security in Android. It makes sure the app you’re installing is valid and at least from the same source as the one you already had installed. For example, if you install, say, my Lockscreen Widgets app from the Play Store, you can be reasonably sure that I’m the one who signed it and that it’s authentic. If you then try to install an update to Lockscreen Widgets from some shady third-party site and it fails, you’ll know that someone tampered with that APK, possibly to add malware.
The key used to sign an app is (ideally) never publicly released. This is known as the private key. The private key is then used to generate the key shown in the app’s signature, known as the public key. This is what Android and app stores use to verify an app’s validity. I won’t get into how exactly you can generate a public key without exposing the private key, since it involves a lot of encryption math. If you want more details, check out Google’s documentation on signing APKs or do some research on one-way math functions.
Another feature of app signatures is the ability to restrict permissions only to apps with matching signatures. Android does this internally for a lot of functions, where only apps signed with the same key as the framework can access certain features.
So now that we’ve given a quick overview of APKs and signatures, let’s talk about App Bundles. This is where APK resources come in. Resources are things like layouts, images, audio, etc. Basically, they’re anything that isn’t code. To better support different display configurations and different languages, developers can make multiple versions of the same resource that are used depending on the device and language.
But in an APK, all of those resources exist, no matter which you use. And they take up space. Depending on the complexity of your app, there could be a lot of unused resources for a lot of devices. This is what App Bundles are made to solve. Developers can generate an App Bundle just like an APK, and that App Bundle can then be uploaded to the Play Store, just like an APK can.
Google then uses that App Bundle to generate a whole bunch of different APKs for different device configurations. Each App Bundle only contains the resources needed for that configuration. When a user goes to download that app, they’re served the generated APK that matches their configuration. This helps to reduce both app download and install sizes, saving bandwidth and storage space.
Of course, installing an APK specific to your device means it’s harder for you to just copy it to another device and install it without issue. Depending on your perspective, this can be a good or a bad thing. On the one hand, it makes piracy more difficult, since users don’t have the whole app anymore. On the other hand, it makes legitimately archiving apps more difficult, for the same reason.
Since Android App Bundles aren’t APKs, you can’t just open an AAB file and install it directly onto a device. When you upload one to the Play Store, Google uses the bundle to generate different (unsigned) APK files. Those APKs have to then be signed before they can be installed.
Instead of asking the developer to sign and reupload those generated APKs, Google instead manages the signing itself. The Play Store either uses a new key it creates or asks the developer for the key they use to sign APKs. With either option, Google handles the public signing for the developer and provides an upload key. Google uses the upload key for internal verification and makes sure the App Bundle (or APK in some cases) the developer is uploading is the right one.
If an upload key is compromised or lost, developers can request a new one, and the signing key used to distribute the app remains unchanged.
There’s a lot more to App Signing, but this is what’s relevant to this article. If you want, you can read more about App Bundles and App Signing on this Medium article by Wojtek Kaliciński.
In theory and in practice, App Bundles are pretty great. They reduce data usage and install size, all without the user having to do anything. But because of how it’s implemented, some developers and security researchers in the past few months have raised concerns. Before I sum up these concerns, I want to take a moment to say that most of what’s written below is directly based on a series of articles by developer Mark Murphy of CommonsWare. You should absolutely check his articles out, since they provide more details and criticisms from the perspective of a developer.
In the classic distribution model, a developer keeps the key they use to sign an APK private. It’s never shared to the public and only authorized people should have access to it. This ensures that only those people can generate a valid APK.
But if you use App Bundles on the Play Store, Google is the one managing the key that signs the APKs users receive. The default behavior for new apps uploaded to Google Play starting August 2021 is for Google to create its own distribution key which it keeps private from the developer.
Developers submitting new apps will have Google manage their private key for them by default, though developers submitting updates to existing apps can continue using APKs or they can switch to AAB distribution by generating a new key for Google to use for new users. Existing apps aren’t required to switch from APK distribution to Android App Bundles, though that option is available to them should they choose. After some pushback, Google will even make it possible to upload your own private key for Google to sign with, for both new and existing apps. None of these situations are ideal, as no matter what, Google will have access to your private key if you want to use Android App Bundles (and developers have no choice in the matter if they want to submit a new app after August 2021!)
While we’re confident that Google takes security very seriously, there’s no company on Earth that’s immune from data breaches. If the key Google uses to sign your app for distribution is in one of those breaches, then anyone can sign a version of your app and make it look like it was signed by you. And some developers and security experts aren’t happy about this possibility. It’s a very, very slim possibility, yes, but the fact it’s a possibility at all scares some in the infosec community.
“Having developers sign Android APKs means anyone can verify APKs from Google Play, blind trust is not required. It is an elegant design that provides verifiable security. App Bundles turn that on its head, and seem structured to promote vendor lock-in. There are many alternate technical approaches that would provide small APKs still signed by developers, but these would not preference Play. For example, all of the APK variants could be generated and signed by the developer, then uploaded to any app store. ”
There are certainly arguments to be made about whether it’s better to leave the secure storage of private keys in the hands of Google or individual developers. But those developers (probably) aren’t usually using a central repository for their keys. By forcing developers to use Play App Signing, a malicious attacker only needs to breach Google’s security once to retrieve thousands or millions of keys.
For what it’s worth, here’s what Google says about how it protects your signing key on its infrastructure:
“When you use Play App Signing, your keys are stored on the same infrastructure that Google uses to store its own keys.
Key access is governed by strict ACLs and tamper-evident audit trails for all operations.
All artifacts generated and signed with the developer’s key are made available to you in the Google Play Console for inspection/attestation.
Furthermore, to prevent key loss, we make very frequent backups of our primary storage. These backups are strongly encrypted and we regularly test restoring from these backups.
If you want to learn about Google’s technical infrastructure, read the Google Cloud Security Whitepapers.”
As great as that all sounds, loss and theft are still possible. And audit trails only help prevent future attacks; they won’t get breached keys back.
Potential for Unauthorized Modifications
One big issue with the way Google has set up App Bundles is the potential for unauthorized modifications to be added to an app. The process of extracting APKs from an App Bundle inherently involves modifications, since Google has to manually build each APK. While Google has promised that it does not and will not inject or modify code, the problem with the App Bundle process is that it has the power to do so.
Here are a couple examples of what a company in Google’s position has the power to do:
Say there’s a secure messaging app that people use to communicate without the risk of government surveillance. This could be an incredibly useful tool for people protesting an authoritarian government, or even people who just want to maintain their privacy. That government, wanting the ability to see what app users are saying, could try to coerce Google into adding a surveillance backdoor into the app’s code.
This example is a bit more innocuous, but it’s also something that concerns some people. Say there’s an app that gets millions of downloads a day, but it doesn’t have any ads or analytics in it. That’s a huge data source with no way to access that data. Google, being an advertising company, might want to access that data.
In the classic APK model of app distribution, Google can’t modify the apps without changing the signature. If Google changes the signature, especially on a popular app, people are going to notice because the update won’t install. But with App Bundles and App Signing, Google could silently inject its own code into apps before distributing them. The signature wouldn’t change because Google would own the signing key.
To be clear, these examples are incredibly unlikely to happen. Google tends to simply pull out of troublesome markets altogether, rather than adapt. But even though it’s unlikely, it’s still possible. Just because a company promises something won’t happen, it doesn’t guarantee it.
Google, hearing these concerns, this week introduced a new feature called Code Transparency for App Bundles. Code Transparency allows a developer to essentially create a second signature that’s shipped with the app to users. This extra signature should be created from a separate private key that only the developer has access to. However, there are some limitations to this method.
Code Transparency only covers code. That might seem obvious given the name, but it also means it doesn’t let users verify resources, the manifest, or anything else that isn’t a DEX file or a native library. While malicious modifications to non-code files usually have much less impact, it’s still a hole in the security of the app.
Another issue with Code Transparency is that there’s no inherent verification. For one, it’s an optional feature, so developers have to remember to include it for every new APK they upload. At the moment, it has to be done from the command line and with a version of
bundletool that doesn’t come with Android Studio. Even when a developer includes it, Android doesn’t have any sort of verification built in to check that the Code Transparency manifest matches the code in the app.
It’s up to an end user to check for themselves by comparing the manifest against a public key the developer can provide, or by sending the APK to the developer for verification.
While Code Transparency allows for confirmation that no code in an app is modified, it doesn’t include any sort of verification for other parts of an app. There’s also no inherent trust in the process. You could argue that if you don’t trust Google, you’re probably up to the task of verifying independently, but why should you have to?
There are other issues with the Code Transparency feature, as pointed out by Mark Murphy from CommonsWare. I recommend reading his article for a more in-depth analysis of the feature.
Developer Convenience and Choice
A third (and final for this article) reason some developers take issue with App Bundles is reduced convenience and choice.
If a developer makes a new app on the Play Store after Google begins requiring App Bundles and they choose the default option of letting Google managing the signing key, they won’t ever have access to that signing key. If that same developer then wants to distribute that app on another app store, they’ll have to use their own key, which won’t match Google’s.
That means that users will have to either install and update from Google Play or from third-party sources. If they want to change the source, they have to completely uninstall the app, potentially losing data, and reinstall. APK aggregators like APKMirror will then also have to deal with multiple official signatures for the same app. (Technically, they already have to do this because App Signing lets you create a new, more secure key, for new users, but it’ll be worse for them and other sites when everyone has to do it.)
Google’s response to this issue is to use the App Bundle explorer or Artifact explorer in the Play Console to download the resulting APKs from the uploaded bundle. Similarly to Code Transparency, this isn’t a complete solution. The APKs downloaded from the Play Console will be split for different device profiles. While the Play Console does support uploading multiple APKs for one version of one app, many other distribution channels don’t.
Thus, a lot of the benefits of using App Bundles go away when developers are managing multiple stores, making distribution more difficult. With news that Windows 11 is gaining Android app support thanks to the Amazon Appstore, some believe that the App Bundles requirement will disincentivize developers from distributing on Amazon. Of course, Google’s primary concern is with its own app store, but that’s exactly what landed them in hot water with competitors leading them to make small, conciliatory changes to how third-party app stores work on Android.
Relevant: Epic’s direct distribution of Fortnite on Android is via APK. And Microsoft’s new Windows 11 support for Android apps is based on APKs.https://t.co/2stinObIsf
— Tim Sweeney (@TimSweeneyEpic) June 30, 2021
A couple related issues to multiple stores are app interconnectivity and rapid-fire testing.
Let’s start off with app interconnectivity. Have you ever downloaded an app that locks features behind a paywall? Almost definitely. Some developers put the features behind an in-app purchase, but others may choose to make a separate, paid, app. When that add-on app gets installed, the main app’s features are unlocked.
But what prevents someone from just installing the add-on from a pirate source? Well, there are a lot of options for developers, but at least one involves using signature-protected permissions. Say the main app declares a signature-protected permission. The add-on app then declares that it wants to use that permission. Ideally, the add-on app will also have some sort of license verification functionality in it, that connects to the internet to make sure the user is legitimate.
If both apps have the same signature, Android will grant the permission to the add-on app and piracy protection checks will pass. If the add-on app doesn’t have the right signature, the permission won’t be granted, and verification will fail.
With the classic APK distribution model, a user can get either app from any legitimate source and be done with it. With current default App Bundle model, the signatures on the main and add-on apps won’t match. Google’s going to make a unique key for each app. The developer could always do away with the signature-protected permission and use direct signature hash verification, but that’s a lot less secure.
And then there’s rapid-fire testing. Users email developers all the time about issues in their apps. Sometimes those issues are simple fixes: reproduce the issue, find the problem, fix it, and upload a new version. But sometimes they aren’t. Sometimes developers can’t reproduce an issue. They can fix what they think is the problem, but then the user has to test it. Now assume that user installed the app through Google Play.
With the APK model, a developer can change some code, build and sign a new APK, and send it off to the user for testing. Since the signature of the test APK matches the one the user has installed, it’s a simple process to update, test, and report back. With App Bundles, this falls apart. Since Google signs the APK the user originally installed, it won’t match the signature of the APK the developer sends. If this app is published after the App Bundles deadline, the developer won’t even have access to the key Google uses. In order to test, the user would have to uninstall the current app before installing the test version.
There are a bunch of problems here. First, there’s inconvenience, both on the developer and user side. Having to uninstall the app just to test a fix isn’t fun. And what if the problem goes away? Was it the changes the developer made, or was it because the user effectively cleared the app’s data? The Play Store does have Internal Testing, which is supposed to let developers do rapid-fire builds and distribution, but it requires the user to uninstall the release version first. It doesn’t really fix anything.
In case this all sounds like a bunch of hypothetical nonsense, here’s a very real example of a developer who will have these problems if they let Google generate a private key for them: João Dias. He’s the developer of Tasker, along with a whole bunch of plugin apps, including the AutoApps suite. With the new App Bundles requirement, João’s development cycle may get a lot trickier, at least for new apps. Sending testing versions directly will be less convenient. Verifying licenses will be less effective.
This may sound like a bit of an edge-case, but it’s not like João is some small developer, and it’s likely he’s not alone. There are many apps on the Play Store that rely on signature verification to detect illegitimate users.
Of course, with the new option for developers to upload their own signing keys to Google, these issues are at least somewhat alleviated. But developers have to opt-in to enable the option for each app. If they don’t, interconnects will fail and rapid-fire support will require uploading a Bundle to Google and waiting for APKs to be generated, before sending the correct one to the user. Plus, it still means they have to share their private key, which brings us back to the concerns we discussed earlier.
This is an old issue given the App Bundle requirements were publicized months ago, so there have been quite a few solutions proposed in the interim.
One solution is to avoid the need for Play App Signing. Instead of generating an App Bundle that Google then processes into APKs and signs, that processing could be done by Android Studio. Then, developers can just upload a ZIP full of locally-signed APKs for each configuration that Google would have generated.
With that solution, Google wouldn’t need access to developers’ keys at all. The process would be very similar to the classic APK distribution model, but would involve multiple, smaller, APKs instead of just one.
Another solution is to just not require the use of App Bundles and continue to allow developers to upload locally-signed APKs. While App Bundles may be a better experience for the user in many cases, some apps don’t actually benefit from being split up per-configuration, with minimal size reduction.
If Google implemented both of these solutions, then a developer who wants to use App Bundles won’t have to hand over signing to Google, and a developer whose app won’t benefit much from the format won’t have to use it at all.
When they were first asked about allowing developers to handle the signing for App Bundles, Google’s response was very noncommittal:
“So, I talked briefly about the requirement next year for new apps to use app bundles, and one thing that comes with that is that by extension we will require Play App Signing. So developers will need to either generate the App Signing key on Play or upload their own key to Play… because that’s a prerequisite for app bundles. We’ve heard from developers that some of them just don’t want to do it. They don’t want to have keys managed by Play. And currently that’s not possible if you want to use app bundles.
But, we’ve heard that feedback, and… I can’t talk about anything right now, we don’t have anything to announce, but we are looking into how we could alleviate some of these concerns. It doesn’t necessarily have to be allowing to keep your own key while uploading bundles. We’re looking into different options. We just don’t have a solution to announce right now. But, we still have around a year until the requirement, so I’m really hopeful that we’ll have an answer for developers for this.”
That was in late November last year, and nothing seems to have happened. With only a few months left before the App Bundles requirement goes into effect, there still isn’t a way for developers to handle signing their own apps. While Google has now made it possible to upload your own key for both new and existing apps, this still takes the signing part out of the hands of the developer.
While Google has specifically promised that the Play Store isn’t going to modify app code, a promise isn’t a guarantee. With App Bundles and App Signing, there’s no technical limitation that we know of preventing Google from modifying uploaded apps before distribution.
Google has introduced Code Transparency as an optional feature, and while this helps somewhat, it has its fair share of problems, as we discussed earlier.
When Google was asked about allowing developers to make their own app “bundles” (ZIPs containing split APKs), the response was basically “we’re not going to do that”:
“Probably not as it’s described in the question, as this would make the publishing process even more difficult for developers, and we actually want to make it simpler and safer. However, again, we’ve heard this feedback, and we will be looking into options how to make this possible, however probably not in the way that was described here.”
Interestingly, Google’s justification seems to be that it would make publishing more complicated. However, Google could still make the process automated as part of the APK generation dialog in Android Studio. Furthermore, if the app in question is being distributed on multiple stores, it would actually make the publishing process simpler, since developers wouldn’t have to manage multiple signing keys and complaints from users.
And with the introduction of Code Transparency, it seems that complication isn’t exactly an issue after all. Code Transparency, for now at least, requires the developer to use a command-line tool and for users to explicitly verify the validity of the app they’re served. This is more complicated than a process to self-make bundles, and it’s unclear why this is the solution Google prefers.
App Bundles will be the required distribution format for new apps submitted to Google Play starting August 1st. While Google has at least somewhat addressed most of the issues raised by developers and security experts, the responses leave a lot to be desired. There are many obvious benefits to App Bundles as the next-generation distribution format, but there will always be lingering concerns with giving partial or total control of app signing to Google.
Google’s responses and efforts are certainly appreciated, but some, like Mark Murphy, feel they haven’t gone far enough. With solutions like self-made bundles not being implemented and the deadline for Android App Bundles being required fast approaching, it doesn’t look like developers on Google Play will be able to retain full control of their apps for much longer.
We’ll be talking about the implications of the Android App Bundle requirement in a Twitter Space later this afternoon, so join us!
Join XDA at 5PM ET on July 1st for a discussion about what’s happening to APKs, sideloading, Android App Bundles, and more!
— XDA (@xdadevelopers) July 1, 2021