Adventures in Gradle Dependency Resolution
Currently at Yelp I'm working on building out our next generation experimentation infrastructure, a large part of which involves writing a Java wrapping library around a Rust library which does most of the heavy lifting of our experimentation. Yelp is mostly a Python shop, but we also have a few backend Java services; this meant that when we went to implement our Java wrapper, we decided to just write it once in pure Java and write a thin layer of Android code on top of that to provide Android-specific capabilities like logging to adb.
This means we have a single Gradle project which is built something like this:
Since our Java code communicates with our Rust code over FFI, we can't just pass configuration objects into our Rust library to control its initialization Instead, we need to pass the desired configuration to the Rust library as a JSON string, and then let Rust create the necassary models on its side of the FFI boundary. Unlike Android, which provides the org.json APIs by default, Java does not come with JSON capabilities built-in, so we needed a library which could create JSON strings for us. We took a quick look at GSON and Moshi, but were both way heavier than we needed for our purpose. Instead, we decided to use Google’s json-simple library, which exposes a very similar API to the one that Android exposes.
From there, library development went swimmingly until we went to add the
latest version of our Android library to the Yelp app. When we did that, our
app would fail to build with the following error:
Warning: Conflict with dependency junit:junit. Resolved versions for app (4.10) and test app (4.12) differ.
What the heck is going on here? Digging around in the Android documentation, we discovered that Android creates an instrumented version of your app for testing, and will yell at you if there are any difference between the instrumented version of your app and the normal production one. This is super helpful, as it prevents you from say, using OkHttp 3.5 in your app and relying on some behavior there, but testing against OkHttp 3.3, where that behavior doesn't exist.
We started bug hunting by first looking in both our library code and app code to see where JUnit 4.10 was coming from. However, we quickly found that we declared JUnit 4.12 for all of our compile configurations in both our Java wrapper and our Android application. What's more, we reasoned that Gradle's default dependency resolution strategy should always choose JUnit 4.12 even if we did let 4.10 slip in as a dependency.
After scratching our head for a while, someone on our team had the idea to look at the pom.xml for json-simple (Gradle uses Maven's pom.xml format for resolving dependencies, even though it is a separate build system) and discovered that JUnit is declared as a dependency of json-simple. This meant that when we pulled in json-simple as a implementation dependency, we were also packaging all of JUnit 4.10 with our app! Then, when Gradle went to build our app, it was finding a conflict between the version of JUnit we accidentally dragged in and the version we cared about for our tests.
The solution to this was to include a little snippet in our app’s build.gradle file.
allprojects {
configurations.all {
resolutionStrategy {
eachDependency { DependencyResolveDetails details ->
switch(details.requested.group) {
case 'junit':
details.useVersion junitLibVersion
break
}
}
}
}
This snippet basically tells Gradle “anytime someone specifies a JUnit dependency, ignore the version that they set and use junitLibVersion (in our case 4.12)”. For more about ResolutionStrategies, you can check the Gradle docs.
After we added this snippet, we could build and test the app. Huzzah! However, we hadn't solved the root cause of our problem (JUnit being packaged with our production app), so another snag came up when we went to merge to master and create a release build. All of our tests passed, but the our release apk build step failed with the following errors:
Warning: there were 5 instances of library classes depending on program classes.
You must avoid such dependencies, since the program classes will be processed,
while the library classes will remain unchanged.
(http://proguard.sourceforge.net/manual/troubleshooting.html#dependency)
When we went to the link, we saw the following description of this error:
In Android development, sloppy libraries may contain duplicates of classes that are already present in the Android run-time (notably org.w3c.dom, org.xml.sax, org.xmlpull.v1, org.apache.commons.logging.Log, org.apache.http, and org.json). You must remove these classes from your libraries, since they are possibly inconsistent, and the run-time libraries would get precedence anyway.
"Libraries that use org.json" sounded a lot like us at first glance, especially since json-simple's package is awfully similar to org.json. However, json-simple has its JSONObject class packaged as org.json.simple.JSONObject whereas Android packages this as org.json.JSONObject. Java uses fully qualified names in order to resolve classes, so we determined that this warning was not due to duplicate JSON packages. After a bit more head scratching, we then went back to the initial console output and saw the following:
11:27:32 Warning: library class android.test.AndroidTestRunner extends
or implements program class junit.runner.BaseTestRunner
11:27:32 Warning: library class android.test.InstrumentationTestCase extends
or implements program class junit.framework.TestCase
11:27:32 Warning: library class android.test.InstrumentationTestSuite extends
or implements program class junit.framework.TestSuite
11:27:32 Warning: library class android.test.suitebuilder.TestSuiteBuilder$FailedToCreateTests
extends or implements program class junit.framework.TestCase
It turns out that our JSONObjects were not the problem at all, it was JUnit! It turns out that forcing JUnit 4.12 resolved our dependency mismatch between test and real apps, but we were still packaging JUnit with our actual application! This version of JUnit was providing classes that Android already had a copy of, and Proguard was helpfully making sure that we didn't get nondeterministic behavior.
Once we figured this out, the fix was straightforward. We didn’t actually need JUnit at runtime, and we’re already compiling it in for our tests. So we can just exclude all of our JUnit dependencies when we depend on the Bunsen library. This is easily done with Gradle by including:
implementation("com.yelp.library:$libVersion", {
exclude group: 'junit'
})
After that, everything worked beautifully and we merged to master smoothly.
Looking back on this process, a few things stand out as learning experiences for me:
- Take time to read your error messages: It’s so easy to read one line and think you know what’s going on, like when we saw “oh, an issue with org.json”. Before implementing, take the time to read glean as much information as you can from your logs. It’ll make your debugging process go way quicker.
- Gradle isn’t black magic: it’s a tool like any other: Gradle’s syntax can be funky and hard to understand, but it’s still just a whole bunch of code that’s running. Understanding how Gradle resolves dependencies and how its syntax works can go a long way towards solving your build woes.
- Make sure you understand the code you're copying from StackOverflow: There are a lot of solutions to Gradle problems which involve slapping together some poorly understood answer on StackOverflow. This leads to Gradle continuing to feel like Black Magic. Make sure to take the time to understand what this solution that someone suggested is actually doing before adding it into your codebase. One great way I've found to do this is instead of copying and pasting code, take the time to write it yourself.
Discussion, links, and tweets
Hey! Thanks for reading! If you like what you read and want more, you can follow me on Twitter.
Tweet Follow @maltzj