Copyright © 2021 Flare Build Systems, Inc. All rights reserved.

Do not copy or mirror any portion of this text without including this notice in its entirety, with or without modifications.

This document is a real-world iOS + Bazel onboarding guide created for a top mobile company and published “as-is” after minimal redactions and input from other Bazel experts at top software companies.

Authors: 

• Zach Gray (flare.build)

Contributors: 

• Keith Smiley (Lyft)

• Oscar Bonilla (LinkedIn)

• Shawn Chen (LinkedIn)

Introduction

This document is made up of two distinct sections; first we start by mapping out some of the landscape of Bazel & iOS in Q4 2020, identifying some important landmarks and possible routes to successful adoption. In the latter section, we’ll plot a course through the mapped terrain and present the recommended path for building, testing and deploying the iOS projects at <REDACTED> with Bazel.

The background discussion on motivation, pros & cons, etc can be found in the <REDACTED DOC> doc. 

Why Now

• Bazel is a #2 engineering priority company wide <REDACTED DOC>

• The wider Bazel & monorepo onboarding effort <REDACTED DOC> is in progress now

• Broadly speaking, Bazel aligns with the <REDACTED DOC> OKRs and will position the project to successfully achieve the related objectives

• UI Testing is becoming a priority; proper test avoidance will be more important as more expensive tests are added to the project

• Waiting longer increases Level of Effort

  • Case in point, the process of adopting FTL would be based on the Fastlane workflow now; the effort of rolling this out will be repeated when making it work with Bazel

• Get ahead of the inevitable issue of slow build/test and degrading DX as <REDACTED> continues to scale

  • iOS CI runs already take xx minutes

  • Software test cost/duration has been shown to increase quadratically over time according to research from Google, driving the need for more sophisticated tooling 

• Build & test cycle is already slow enough to have inspired the homegrown playgrounds as well as a somewhat complicated custom templating solution to enable Previews

Part 1: Mapping the Terrain

It’s worthwhile to begin this discussion by defining the current state of Bazel+iOS, as this is easily the most challenging area for organizations adopting Bazel; the rules are undergoing big changes frequently and the current cohort of Bazel adopters are upstreaming lots of excellent work. 

Migration Patterns

There’s are various distinct routes available by which to migrate the <REDACTED> iOS app suite:

1: “In Place”

Characteristics

• BUILD files are generated automatically from the existing mechanism, XcodeGen YAML project files. This is a little strange, considering it’s the inverse of what tools like XCHammer do (convert targets in BUILD files -> XcodeGen YAML)

• The target defining the top-level application bundle will either be generated with a dependency on these frameworks, or created and kept up to date manually during the migration. YAML files are the source of truth and there is no change to the existing developer workflow

• Changes to project structure and Swift code are avoided almost entirely

• It may be necessary to create additional AppDelegateBazel shim classes for the duration of the migration so there’s a Bazel-specific entrypoint

• Heavy customization takes place on any of the rules necessary to facilitate the above

2: Best Practices

Characteristics

• If BUILD file generation is implemented at all, it’s done with a reasonable tool like Gazelle which delivers a developer experience on par to what Gazelle gives to Golang engineers. Given the need to define a metaproject somewhere in a way which isn’t really necessary with Golang, BUILD generation is unlikely to be solved entirely by Gazelle alone

• Targets in BUILD files are the source of truth from which all project generation happens

• Developer experience changes to align with a typical Bazel experience

• Changes to module architecture is implemented as necessary to align fully with the expectations of the various rulesets (for example the *_framework targets created by Bazel for use within an app don’t align perfectly with <REDACTED>s approach; additionally, mixed-source modules should be avoided for other reasons)

• Again a shim is likely employed to retain an entrypoint for the bazel-build applications

• No major changes are made to any third-party rulesets (in the context of the workflow and DX)

End States

The following states of Bazel integration are not necessarily mutually exclusive; over the course of a migration to build & test most iOS apps with Bazel it’s common that each of these states will briefly be occupied to some degree. However, it’s important to note that significant additional effort would be required to make each of these states “production ready” and thus it’s recommended to make this investment only for the chosen end state. 

Bazel in CI Only

In this model, the developer workflow remains unchanged but CI builds, tests, and deploys all happen with Bazel. This is the model LinkedIn has been using for a few years, however they’re slowly rolling out Bazel for developers (see rules_ios). 

Under this model, generation of BUILD files for CI based on XcodeGen YAML files is likely necessary, because there’s very little chance developers can and will maintain both the YAML project definitions as well as BUILD files. 

Bazel in CI + Dev (Hybrid)

In contrast to the above, in the hybrid model, Bazel and xcodebuild are both supported for local development. Some tasks such as project generation, perhaps some of the unit test suite are run under Bazel, and xcodebuild is used for indexing and building with debugging. This is suboptimal because developers sacrifice the speed and correctness of Bazel for the developer experience of a fully functioning Xcode. 

Bazel in CI + Dev

This best-in-class outcome for Bazel-built Apple applications is the hardest to achieve but thanks to recent advances in the community this outcome is possible with much less effort than required by Uber, Lyft, Pinterest and LinkedIn as early adopters. In this outcome, both CI and iOS engineers make use of Bazel for 100% of use cases, with the IDE integration working so well that mobile engineers observe minimal or no loss of functionality over the traditional tooling. 

Rulesets

When building large-scale Apple applications with Bazel, there are many distinct rulesets to be aware of and make use of. In the case of <REDACTED>’s iOS and watchOS application, most of these will be relevant.

rules_apple 🔗

Google’s Bazel rules for bundling Apple apps, both macOS and other *OS. Also handles provisioning profiles and the like.

rules_swift 🔗

Support for compiling Swift code to units digestible by rules_apple - note that ObjC code is built by Bazel (it’s an old native rule) and eventually rules_cc as that rolls out.

rules_ios 🔗

Higher-order rules which build on top of all of the above and include the following notable features among others:

• Xcode project generation, created by Square

• Mixed-source modules (Swift and Obj-C)

• Indexing integration

• Better debugging

rules_ios was created initially by LinkedIn as an internal ruleset which evolved into public design documents (focused primarily on implementing mixed-source libraries) and has since been expanded to fill in some of the obvious gaps the rest of the industry outside of Google have experienced with Bazel’s Apple support, with Samuel at Square taking the lead on this effort as well as contributions from various other large iOS teams upstreaming some of the work they’ve done during their adoptions of Bazel for iOS. 

This ruleset is emerging as an excellent starting point for organizations adopting Bazel as it fills in many of the gaps left  by Google’s approach to the rulesets (which resulted in lots of community fragmentation early on) and has lots of functionality landing frequently from the current cohort of iOS@scale organizations. 

rules_apple_line 🔗

This ruleset again solves some of the shortcomings of rules_apple / rules_swift and is maintained by Line. Most of the rules in here are likely semi redundant, but there are a few utilities and helpers such as pkg_dsym that may be worth using, even if as a reference for implementing something similar on top of other rulesets. 

ios_bazel_users

This repository includes other resources contributed by Lyft, Pinterest and others to show common use cases like Xcode indexing, an old approach to support Bitcode under Bazel (a recent addition to Bazel itself) and other topics. 

Packages (BUILD files)

Granularity

Currently within the <REDACTED> application, there are approximately 52 Xcodegen YAML files which map to independent app modules or internal frameworks and produce either static or dynamic frameworks. During the initial phases of migration, each of these YAML files defining AppModules or Internal Frameworks could be migrated to a top level package and one framework (or library; see discussion in <REDACTED DOC>). From here, additional granularity might be implemented as an optimization, first to slow targets and in cases where recursive globs were employed (generally an antipattern, but sometimes a necessity for short-lived targets). 

Generation

The current developer experience at <REDACTED> for iOS today is actually fairly close to what it would look like under Bazel: developers must create and modify targets in YAML files, defining all of their dependencies (even transitives) and at times edit the templates used to define the targets. Additionally, any time a YAML file is changed, the project must be regenerated and the IDE restarted.

BUILD files could be generated from YAML files as part of the in-place migration described above, but these YAML files are somewhat complex and hard to maintain as is, due to the implementation of the templating engine in XcodeGen. 

Tooling like Gazelle could be utilized to generate apple_library or similar targets from Swift and Objective-C code, but there’s a few blockers here:

• As of yet, no one in the iOS community has gotten any use out of Gazelle 

• A few orgs have worked on Xcode -> BUILD file generation, but none of this has landed in any usable manner yet in any public repositories

• It’s not really possible to generate the top-level project metatargets, and in general in the iOS ecosystem even intermediate compilation units are not well suited to generation beyond what the rulesets themselves are doing under the hood

Replacing all of this with BUILD files would likely be a quality of life improvement for developers, even with no code generation whatsoever (outside of buildifier for automation of some tasks, and macros to reduce boilerplate). As such, BUILD file generation is likely a non-goal.

IDE Integration

Critical Functionality

Project Generation

Tulsi 🔗

Google’s bridge from Bazel -> Xcode. Generally speaking, this hasn’t been sufficient for almost anyone outside of Google doing things at scale; however, some pieces of it are quite useful, and some smaller organizations have quietly gotten some mileage out of it. For example, we’ve successfully built Kickstarter’s iOS application under Bazel with a Tulsi workflow--but it leaves a lot to be desired. 

XCHammer 🔗

Created by Pinterest, XCHammer combines Tulsi’s aspect and XcodeGen to generate Xcode projects. This approach is universally accepted as the correct way to go; sometimes making use of queries instead of the aspect, but for the first cohort of adopters (Uber, Lyft, LinkedIn, Pinterest) the general shape of the thing is often something like this with the exception of rules_ios. Trivia: for a few reasons, Lyft rolled their own solution here that is very similar, but primarily because there was no good alternative at the time.

xcodeproj() via rules_ios

The rules_ios ruleset provides a rule which handles Xcode project generation.

Indexing

Enabling Xcode indexing is a hard requirement for features like jump to source and autocomplete in the IDE. At a minimum, Xcode needs to interact with Bazel’s output to properly index (in the case of objc; it can typically accomplish the live-indexing via swiftc alone in Swift projects). Ideally, indices output by Bazel during the build are sufficient and no additional indexing is required; this is the ideal state. It’s worth noting that in the case of pure-Swift projects, it’s possible to rely on Xcode’s default behavior, while using a run script to handle the invocation of Bazel; this is what Lyft did. Unfortunately that is unlikely to work as easily at <REDACTED> due to mixed sources. 

There are many paths to implement this functionality, but at this point it’s unlikely there’s a need to reimplement it from scratch. 

Indexing Implementations

• Lyft’s index-import works with rules_apple and rules_swift. For the the tool to work, bazel-out needs to be synced to DerivedData which is the default search path for Xcode

• As of a few months ago, rules_ios provides some indexing capabilities with it’s xcodeproj rule 

• Within the iOS community it’s fairly well documented how to create and consume Xcode indexes and various other companies have created their own in-house tooling for this, although this is non-trivial and should be avoided if possible

Build & Test

Results

Build & test results should be integrated back into Xcode, as well as runnable via Xcode’s build and run commands. This is the bare minimum level of functionality users will expect. This is possible to achieve with really any approach, whether that be rules_ios, something custom, Tulsi, etc.

UI Integration

Tests should be collected and displayed in the Tests UI and runnable via the “diamond” start buttons.

Tests

Unit Tests

Unit tests in <REDACTED> are all XCTest-based and as such fairly straightforward to handle with both rules_ios and rules_apple again with the expectation that for mixed sources, rules_ios, rules_apple_line, or custom rule implementations will be necessary.

UI (Integration) Tests

Similar to unit tests, UI tests within <REDACTED> are also based on the standard ui-testing mechanism and as a result, rules_apple should be sufficient here.

Test Runner

Fastlane is used to run tests in CI; this functionality will be fully replaced by Bazel and likely without issue. See Fastlane for more discussion, and also make note of the related discussion on Bluepill.

Code Coverage

At <REDACTED>

iOS code coverage metrics at <REDACTED> are currently collected and observed, but a specific goal is not set or enforced. The team plans to use some of this data in automated code review bots, and this is functionality they’d like to keep.

Under Bazel

iOS code coverage under Bazel is still a bit of a challenge because most of the support for this in the rules is Google-internal only. However, various other members of the community have at least worked through this by now and shared some of their learnings. This is not something that will work 100% “out of the box” but it is still easily accomplished in a few days of effort. 

A good starting point for implementing coverage reports can be found here.

1st-Party Dependencies (Modules)

There’s currently an in-progress effort to modularize and decouple modules within <REDACTED>. Each module has a corresponding XcodeGen yaml file and generates a dynamic framework or static library conditionally.  For the most part, it should be possible to map these existing modules to the corresponding framework rule, but there will likely be some subtle differences worth discussing. 

Additional background information on dynamic frameworks can be found in the Links & Resources section of this document.

How it works today

The iOS team already makes use of a workflow similar to what building the iOS app under Bazel might look like:

1. Open the workspace in VSCode or similar to edit YAML files defining the project structure

2. Invoke make project or make project-preview to generate and open an Xcode project from the YAML and source files

3. Standard Xcode usage

The above make commands invoke a script that runs over the YML files and expands templates (before XcodeGen’s templating engine itself is engaged). This script either generates static libraries for defined Library template targets when running make project, or dynamic frameworks when expanding for the Xcode Preview use-case via make project-preview. This is done because Xcode Preview requires use of dynamic frameworks. There are 16 AppModule targets and 13 InternalLibrary targets which make use of this codegen functionality to generate either static library or dynamic framework targets.

In addition to the above, there are also 28 dynamic frameworks within <REDACTED>, and there is also 1 fully static library - ServerDrivenFlow (although it’s not clear if the template overrides this hard-coding of type: library.static; perhaps this should be checked).

Dynamic frameworks are necessary in cases where App Extensions will be depending on the internal frameworks and multiple modules will be depending on the same framework; if it were statically linked it could be linked into the final binary multiple times. 

For reference, guidance on internal modules within <REDACTED>’s iOS application can be found here.

How it could look under Bazel

As described here, rules_apple makes the assumption that for the most part, “dynamic” frameworks are the preferred (only) mechanism for building internal modules for use within an application, while static frameworks are mostly bundling rules meant to package things up for publishing. While adopting this behavior may not be an issue, remember that rules_apple doesn’t support mixed-source modules at all and so it’s a non-starter to rely on this construct directly unless mixed-source modules are pre-factored into separate modules - while recommended, this is considered to be an optimization and as such staged for that phase.

In rules_ios, while it’s built on top of rules_swift and the objc support coming from Bazel, its implementation of frameworks is static only. Presumably, this behaves in the same manner as Xcode and correctly handles the case of multiple dependents, but this should be verified in the initial spike.

rules_apple_line also defines mixed-source framework rules which also create static frameworks meant for internal use, and correctly supports transitive dependencies but defers linking them, similar to Xcode. 

If <REDACTED> were to make use of either of the two rulesets supporting mixed-source modules, it’s possible that apple_library and additional shared resource bundles would be the analog to <REDACTED>’s various Library concepts where dynamic frameworks are needed; however, if the static libraries are linked intelligently liked as advertised, these rules may also be sufficient. More work is needed to determine if this is a viable or advisable approach and to understand what the project structure implications might be in the context of <REDACTED>’s project, and if avoiding the estimated 1 month FTE effort required to remove mixed-source modules. 

3rd-Party Dependencies

Vendored CocoaPods

There are a few legacy CocoaPods dependencies which have been directly vendored into the iOS repository. This will need to have BUILD files created either manually or generated using one of the available tools, PodToBUILD from Pinterest or cocoapods-bazel by Samuel at Square, in collaboration with LinkedIn.

If rules_ios is adopted, it also provides repository rules for building Cocoapods projects easily.

Carthage

Again rules_ios provides what is probably the best approach to handling Carthage dependencies within an iOS application. This is the most important type of dependency as the vast majority of <REDACTED>’s third-party dependencies (26) are coming from Carthage outside the vendored pods. There are limitations to what this rule can do (I believe it supports compiling from sources only) but this repository rule is fairly straightforward to patch and extend.

Publishing & Apple Ecosystem

At <REDACTED>, publishing of the iOS app to AdHoc, TestFlight, Beta, Nightly and Release channels is all handled one again via Fastlane. Fastlane is a big tool which covers a lot of surface area; here’s a summary of how it’s used at <REDACTED>.

Fastlane

Today, <REDACTED> uses Fastlane for a few use cases:

• General CI entry points

  • Build

  • Test

  • Release

    • Upload to app store

    • Download dSYMs

    • Code signing certificate management

Deprecating Fastlane

Recently Apple has released additional APIs that allow easier access to some of these functions. Additionally, rules_apple handles a good portion of what Fastlane does, making it almost redundant. Microsoft reports entirely removing dependency on Fastlane, and Spotify also has mostly moved away from it. 

Functionality to replace

• Uploading of dSYMs to various vendors (Sentry, Crashlytics)

• Upload artifacts to S3

• Managing of provisioning profiles and codesigning; rules_apple handles some of this

• Keychain interactions

• Downloading of dSYMs after uploading Bitcode apps

• Firebase Test Lab

Blockers

The primary challenge is that currently Apple doesn’t offer a way to download dSYMs in their new APIs. Applications uploaded with Bitcode enabled (some iOS apps, WatchOS apps etc) will still need to download these dSYMs for crash log debugging, etc, meaning reliance on Fastlane likely can’t be entirely removed unless Apple makes this API endpoint available. Disabling bitcode is a good option here in some cases!

Migration

During the migration, it’s advisable to keep Fastlane around and integrate it with Bazel via custom rules, with an eventual future goal of totally removing it. Other organizations have replaced Fastlane with a set of backend services which handle all of these use cases in a manner which aligns better with Bazel. 

Fastlane provides a mechanism for invoking individual commands directly, rather than via a Fastfile; this should make wrapping the necessary functionality in Bazel relatively straightforward. 

Fastlane actions to wrap: 

• https://docs.fastlane.tools/actions/download_dsyms

CI/CD

<REDACTED> has already moved iOS builds to Buildkite; running Bazel on the macOS Buildkite workers should be no issue and align with the overall <REDACTED DOC> for the most part.

Bazel needs to run on macOS workers due to licensing restrictions and lack of compatibility for running the Apple toolchains on anything other than macOS; this is ideal however because it’s a necessity to share cache artifacts between CI and developer machines. It is important to align developer Xcode versions and other toolchains closely with CI to maximize cache hit rates if this has not been done, but this is fairly straightforward.

Monorepo

Moving the iOS project to the monorepo is mostly beyond the scope of this document, however it’s worth noting that while moving to the monorepo before or during the Bazel onboarding may incur some additional work, it will also save some effort in other places and prevent duplication by relying on the existing Bazel infrastructure already running in the monorepo.

Remote Features

Cache

There are no real issues or blockers with Bazel remote caching, as long as there are macOS build workers available (which there are via buildkite) then the remote cache could be populated easily with artifacts reusable by iOS developer Macbooks.

Remote Execution

There are significant challenges to remote build & test execution for iOS projects, but it’s possible for some projects (for example, it’s often easier for pure Swift projects). Depending on the topology of the build graph, there are sometimes performance improvements to be had. Some important caveats:

• Remote execution for Apple builds should be run on Apple hardware for a few reasons

  • Terms of use of the Apple toolchains

  • Bazel’s limitation of cross-platform builds

  • The inherent instability and inoperability of most of the options for achieving cross-compilation

• Mixed-source projects which require lots of linking of ObjC and Swift code are challenging to support building remotely due to the use of absolute paths in modulemaps among other things. Explicit Module Builds are a must!

• Linking remotely is still not really worked out yet by the community; those who are experiencing any success (Google, Lyft) are linking locally 

• Some of the required workarounds for debugging remotely built Swift are enumerated here.

• Actions should ideally be executed in a clean VM; without a system like docker, this is hard to achieve using any existing tooling

• Little to no iOS build support by OSS solutions and managed SaaS vendors; note that there’s a difference between macOS CI and build workers and actually fully-functioning iOS applications remotely built on these workers with the results useful to developers

• Remote test execution is even more challenging given the need to run simulators in most cases, but it’s something some vendors (Flare, not sure who else is here yet) are working on

Risks

SwiftUI Previews

Getting this working under Bazel might be considered “advanced” functionality that a few users have had some success with but these have been one-offs that haven’t really been upstreamed yet. This feature may not be truly necessary if the impact of Bazel meets expectations, but if it’s functionality which needs to be retained, a small amount of expert hands-on effort will be required to implement it—though it’s fairly easy with the right skillset, per Keith @ Lyft.

Indexing

Getting this integration wired up has traditionally been a big challenge, and is necessary for any sort of a good developer experience. If existing rulesets don’t meet <REDACTED>’s needs out of the box, significant effort by iOS and Bazel experts may go to this area. 

Mixed Source Modules

This is easily the most complicated and challenging part of building iOS apps with Bazel, perhaps other than (but closely related to) remote build execution. There is a lot of work that has gone into various hacks and workarounds to get header maps and module maps to work correctly under Bazel; rules_ios luckily captures a lot of this but it’s still a risk area.

Something else to note is that mixed-source modules are extremely slow to build in comparison to pure Obj-C or Swift modules; part of the reason Google and now even the new OSS maintainers will never support mixed source modules directly in rules_apple is really because it’s an antipattern to begin with. Google doesn’t have this problem because they stopped using mixed-sources years ago; Square doesn’t have the problem because they refactored. The rest of the community like LinkedIn and Pinterest devised their own ways to handle this after years of work. 

Change to DX

Significant changes to the development experience should be expected; specifically around the generated Schemes and ability to select and run specific schemes easily in the UI; spikes are needed to understand the current state of the various rulesets in play.

Community Fragmentation

Google’s semi-unusable approach to open-source iOS Bazel support has resulted in quite literally all of the initial adopters of Bazel creating their own mechanisms for much of what a successful build ruleset should provide for building large-scale iOS projects. Luckily this is dramatically improving as of recently, but it’s still an area of some risk; it could be the case that heavy modification and customization is necessary to any of the adopted rulesets, but ideally this should be limited to things applicable to the wider community (meaning changes to the <REDACTED> project architecture and practices are incurred). 

Optimizations

Linker 

Linking iOS applications is a substantial portion of the incremental build time and as such, most companies building iOS apps at scale make use of a modified linker which is highly optimized (used in the local development loop only). ZLD is very popular and has significant overlap with iOS Bazel users; rules_apple_line has support for overriding the linker specifically for this, and even an example. While this required a forked version of Bazel in the past, that is no longer the case due to a recent update to the LLVM project addressing a relative pathing issue on that end instead. 

In some cases, ZLD can further reduce incremental compilation times by another 40-50%.

Test Runner

Bluepill

Bluepill is an alternate test runner with a focus on the Simulator; it’s mostly working with Bazel and can net big improvements to test times in CI. This is a good stop-gap given the inherent difficulty in running true remote test execution via Bazel as covered in the relevant discussion.

Bluepill is likely the most relevant test runner to Bazel, while others may be potentially harder to integrate or offer more tangential improvements in other areas.

Firebase Test Lab

Firebase Test Lab is a managed mobile device farm for e2e testing applications on real devices. Traditionally the whole application bundle is uploaded to these services and their invocations are usually nightly or before the app is released; not so much in the diff and premerge check pipeline. While not a drop in replacement test runner, support for this service is still going to be important to implement under Bazel. 

Part 2: Suggested Route

In this section we’ll discuss in detail the prescribed route to adoption of Bazel at <REDACTED>.

Migration Pattern

Make use of the Best Practices migration pattern, making changes to the overall project structure as needed to fully align with what the rest of the iOS Bazel community is doing wherever possible.

Rationale

Given that <REDACTED>’s iOS codebase is still in the early stages of explosive growth, many of the current development practices are not necessarily as hard to change as they were for some other well-known iOS Bazel users who brought with them a decade’s worth of ObjC and hundreds of person-years in existing application and infrastructure code. 

There is even likely room for improvement in a few areas:

• From a regulatory point of view, much of what Bazel offers (reproducibility, correctness, increased automation) should serve to improve upon the current story if anything

Aligning project structure and development & CI practices with the wider Bazel community will allow use of existing rulesets as is with minor modifications, meaning upgrading as new functionality becomes available is low cost. Additionally, extensions to these community rulesets can potentially be contributed back upstream as they are likely to be applicable to the community at large.

The characteristics of this pattern are defined in section 1 above.

End State

Employ Bazel in CI + Dev; rely on Bazel for both CI and local development use cases, invoking all build, test, and bundle tasks with Bazel and related tooling. Achieving this goal will require all Critical Functionality for the Xcode to be implemented, as well as a solution for Publishing and Apple-related tasks (likely a wrapper around Fastlane to begin with).

Rationale

This is the best-in-class outcome for iOS projects making use of Bazel. Achieving this allows for further massive improvements to the DX for iOS at <REDACTED> by enabling the advanced Remote Features to bring big speed increases to the application development teams daily routine. 

Intermediate States

As the Bazel implementation progresses, <REDACTED> iOS will temporarily be in some of the other documented states; for example, it’s likely Bazel will first be deployed as CI-only as discussed. The properties (and difficulties) of these stages of maturity is well documented.

Rulesets

Make use of rules_ios in addition to the baseline rulesets it builds upon (rules_apple and rules_swift). Necessary changes and improvements should be discussed with the community and upstreamed where possible. Refer to the discussion of this ruleset in Section 1. 

Evaluate multiple rulesets with spikes as they have all seen dramatic changes in the recent months and in general this is typically advisable in the initial stages of Bazel adoption, as every iOS project at scale tends to have idiosyncrasies which lend themselves better to a particular method of integration. For example, some of the work in rules_apple_line may be applicable, or perhaps the Project Generation in rules_ios doesn’t align with <REDACTED>, etc. 

Rationale

As called out in the discussion, there’s a hard requirement on one of these rulesets (or something custom-built) to build the mixed-source modules within <REDACTED> until they are removed. Building from-scratch support needed for this and other functionality (or even heavy departure from the ruleset expectations) is bad both for <REDACTED> and the community as it contributes to further fragmentation and should be avoided.

Packages

Start with module-level package granularity as this maps cleanly to the existing project structure. Where necessary, granularity can be increased on a case by case basis but generally this should be deferred to the optimization phase.

Code generation should be considered an optimization, and even in this context, it’s not considered necessary; potentially not worth the significant investment necessary. Revisit the relevant explanation if necessary.

IDE Integration

Implement full Xcode integration as this is a necessity given the chosen migration pattern. 

• Enable Xcode project generation via rules_ios, falling back to other mechanisms only if necessary

• Ensure Indexing via rules_ios works within the <REDACTED> project, and to a suitable degree (test UI working, etc). If necessary, go hands-on with either upstreamable patches or even a manual indexing implementation in the extreme case

• Build and Test commands (keyboard shortcuts) in Bazel should work, however exactly what this is running will likely change, as a spike is needed to understand what targets and schemes will look like under Bazel and the selected rulesets

• Build & Test results should feed back into the Xcode UI where applicable

Tests

Create test targets for unit and integration tests

via rules_apple and rules_ios, replacing Fastlane as the entrypoint invoked by CI in addition to integrating with Xcode and the local development experience. 

Ensure test results from CI are cached and reused on macbooks.

Implement Test Coverage

This is a more advanced topic and a good candidate to move to the optimization phase if the scope of the initial migration becomes onerous.   As we’ve seen, in most cases this should be fully supported by the existing rulesets.

Integrate deeply with advanced test tools as they are rolled out at <REDACTED>, providing first-class support under Bazel:

• Firebase Test Lab

• Flank 

• EarlGrey

• eDistantObject

1st-Party Dependencies (Modules)

Retain the current module structure as much as possible. 

• Create additional modules as necessary to avoid circular dependencies, a risk called out in Section 1.

Adhere to the granularity guidelines defined previously.

This topic is described in detail in Section 1.

3rd-Party Dependencies

As covered in the first section, there should be no significant issues with 3rd-party dependencies within the <REDACTED> iOS project.

Make use of rules_ios to handle Carthage dependencies;

if this ruleset is determined to be suitable for the wider project during the evaluation spike, evaluate use of the relevant repository rule at a minimum.  

Employ rules_ios for the vendored CocoaPods;

consider getting them from the remote repositories instead of vendoring.

Publishing & Apple Ecosystem

Fastlane is a sprawling tool which does a lot at <REDACTED>; as such it’s a tool that will need to be initially relied upon before being phased out of use only after the initial migration.

Wrap key Fastlane functions in Bazel

As covered extensively in the discussion, Fastlane actions can be invoked in a standalone manner without a Fastfile; this should integrate reasonably well with the rest of the build running under Bazel, and Lyft may even have some insights to share here.  Note that during the optimization phase, this work should be removed. 

Implement other Fastlane functionality which cannot be wrapped

for example, Fastlane plugins probably make more sense to implement directly. 

Callout: Firebase Test Lab support would likely have come along as a Fastlane function to duplicate if it were widely rolled out at the time of writing.

CI/CD

CI machines will invoke Bazel for all tasks as covered in the relevant discussion. 

CI will submit applications for e2e testing on Firebase Test Lab; this is a newer addition to <REDACTED>’s suite of tools that will be rolling out shortly.

Remote Features

Deploy a shared remote cache

during the initial rollout. 

CI workers will have read/write access and developer macbooks provisioned as readonly

This will prevent cache poisoning and bloat from bad dev machine configurations, but it’s important to monitor hit rates closely and troubleshoot issues as they arise.

Remote build execution

will be evaluated during the optimization phase only, due to the inherent complexity and lack of a guaranteed RoI.

Remote test execution

will be evaluated separately from build execution as it’s potentially even more of a challenge. 

The challenges of adopting remote features are once again discussed in greater detail in the preceding portion of this document.

Success Measurement

Align with DevX Success Measurement

as defined in <REDACTED DOC>

• CI Build time

• Build stability (master pass rate)

Integration with the existing Build Metrics dashboard

• Identify true apples-to-apples comparisons

• Properly align metrics to maintain continuity in graphs

Optimizations

Following the initial migration phase, additional optimizations should be pursued. A non-exhaustive list of recommended items is included below.

Linker

ZLD should be evaluated for adoption during the optimization phase; either via adoption of rules_apple_line or a patch applied directly to rules_apple; the benefits of this are covered in detail in section 1.

Test Runner

Run a spike to evaluate Bluepill during the optimization stage as it may bring significant benefits. 

Increase Granularity

Fine-grained intermediate library targets and packages should be evaluated. Increasing package granularity beyond the initial state of module-level will increase incrementality and cacheability of the iOS project, as well as increase performance and potentially decrease complexity of BUILD files, as it’s likely that slow and often confusing recursive globs were employed heavily during the Bazel migration. Additionally, these fine-grained targets will enable more parallelization if remote build execution is achieved. 

Remote Build Execution

Evaluate remote build execution for iOS via Flare or other vendors if/when they offer it. As discussed in Section 1, there are significant challenges to this, but a spike should be run to explore the topic in depth in the context of <REDACTED>’s application as the potential build and test improvements are vast.

Remove Mixed Source Modules

As discussed, mixed-source modules are suboptimal and should be avoided. Given the effort this will involve from the application team, this is determined to be an additional optimization which can be undertaken after the success of the initial migration. 

Completely Remove Fastlane

We’ve covered the necessity of Fastlane in the initial stages of the Bazel rollout, however this is something that should be moved away from in the long-term as it’s mostly redundant and adds complexity. 

Implementation Plan

This speculative implementation plan outlines 7 high-level phases of implementation. 

1. Spike: Rules

Success Criteria

• Third-party dependencies built with Bazel

• The simplest single <REDACTED> app module with no 1st-party dependencies built & tested under Bazel

  • Build with Bazel and link to 3rdparty deps

  • Test with Bazel

  • Build/Test locally a single developer machine

  • Verify output is comparable with that of xcodebuild

• Evaluate and document any changes needed to project structure; scope out necessary changes

• IDE Integration validated as likely to meet requirements; enhancements planned & estimated

Epic(s)

1. Spike usage of rules_ios to complete all of the goals stated above

2. Spike rules_apple_line only if necessary (rules_ios is inadequate)

3. Spike custom rules only if necessary (rules_apple_line, rules_ios are inadequate)

2. Spike: CI & Tools

Success Criteria

• The module from Phase I built & tested in CI

• The module consumed by dummy applications (watchOS, iOS)

• Bazel artifacts shared from CI with developer macbooks, increasing local build performance

• The Bazel-built dummy application & module packaged and deployed via AdHoc distribution

• A rudimentary Bazel CI pipeline is in place but not required to pass before landing diffs

Epics

• Consume the module from Phase I in dummy apps built and tested with Bazel

  • Unit tests run for module

  • Basic integration tests run for dummy apps

    • FTL integration

• Spike out usage of Bazel in the iOS CI pipeline

• Spike out Fastlane wrapper for packaging and deployment

3. Migration: Dev Tooling, Modules

Success Criteria

• Spiked items from phase 1 & 2 shored up

  • Action items from spike phases acted upon

• DX is solidified (IDE integration, Bazel interaction etc)

  • Close to end state

  • Nice-to-have functionality deferred to optimizations

• All modules (App Modules, Internal Frameworks) built and tested by Bazel locally and in CI

• All tasks executed in parallel; the module migration teams are not blocked by tooling team

• Legacy build system still in place and functional

Epics

• Execute IDE integration action items from Phase 1 & 2 (TBD)

• Implement & extend rules, macros as needed

• Migrate modules

  • Land completed modules into main/develop branch directly if possible

    • Spin up an integration branch only if restructuring is required which will break legacy tooling

  • Migrate modules to build & test with Bazel by traversing the dependency tree from the leaves towards the root

  • Execute module migrations in parallel as much as possible

  • Adhere to the granularity guidelines

  • Define and contribute shared rules and macros 

  • For each module:

    • Define top-level framework or library target

    • Create intermediate libraries where necessary

    • Define top-level test suite target

    • Implement necessary project structure and minimal code changes (circular deps, imports, etc)

  • Include the new module target in CI run

4. Migration: Application

Success Criteria

• The <REDACTED> app is building under Bazel along with all of its dependencies

• <REDACTED>’s iOS app integration test suite runs under Bazel

• The Bazel-built application starts and runs as expected

• Legacy build system still in place and functional

Epics

• Define the top-level application target(s)

• Define the top-level integration test target(s)

• Define additional Apple ecosystem targets

  •  (entitlements, provisioning profile, config, plist, etc)

• Include the application target in CI runs

5. Rollout: CI/CD Build & Test

During Phase 2, basic CI integration was achieved. In this phase, cut over to running the diff pipeline under Bazel as a required premerge check

Success Criteria

• The Bazel CI pipeline is now a required premerge check and has replaced the legacy system for this use case

• iOS application developers are up to speed on enough Bazel to maintain BUILD files with the help of DevX

• Legacy build system still in place and functional for release artifacts only

• The CI pipeline is fully integrated with Firebase Test Lab

• Tooling is in place to enable Success Measurement

Epics

• Integrate the Bazel pipeline with Phabricator/Github PRs as a required premerge check

• iOS team Bazel onboarding

6. Rollout: CI/CD Distribution

In prior spike phases, initial work was done on tooling to replace Fastlane. Building upon this work, in this phase Bazel is rolled out for packaging and distribution of the application to the various channels.

Success Criteria

• All release artifacts are built under Bazel

• Fastlane is wrapped to facilitate anything the rulesets don’t provide

• Bazel-built release artifacts are uploaded to App Store & Ad Hoc channels

• dSYMs are downloaded from Apple for Bitcode enabled uploads

• dSYMs are uploaded to Sentry and Crashlytics

• Legacy build system is no longer in use

• Legacy publishing tooling is no longer in use

Epics

• Execute any deferred action items from Phase 2 (TBD)

• Fully configure production code signing

• Integrate new Apple ecosystem tooling with the Bazel CI pipeline

• Cut over to Bazel for production releases

7. Optimization

Success Criteria 

• Optimizations are stack-ranked and executed based on priority

Suggested Ranking

1. Increase Granularity

2. Implement a faster linker

3. Evaluate other test runners

4. Remove mixed-source modules

5. Deploy remote execution

8. Maintenance

The majority of the effort around keeping the Bazel-related elements of the project operational & up to date will be handled by DevX resources, while product engineers will focus primarily on BUILD files specific to the application modules they’re working on. It is advisable that the DevX team staff a few resources with a strong background in the Apple ecosystem.

Breakdown of Responsibilities

DevX

• Bazel version upgrades

• Bazel configuration, CI configuration

• Rules versions upgrade

• Custom rules, macros (most of what’s in .bzl files)

• Xcode support

  • Note: the community has gotten much better at keeping up with Xcode releases, often updating the rules to support them even during Beta. 

Product Engineers

• BUILD files will be managed by Product Engineers

  • Defining app and test targets and the frameworks they depend on

  • Defining the dependencies between various targets

• Simple macros & genrules created by product engineering leads to reduce boilerplate and DRY up the BUILD files

Links & Resources

Static & Dynamic Frameworks, Libraries

• Dynamic Library Documentation from Apple

• Frameworks as understood by Google in rules_apple

Copyright © 2021 Flare Build Systems, Inc. All rights reserved.