Tune your Bitrise workflows using cache in steps

How can you speed up your workflow using Brew, Gem, and CocoaPods packages cache?Guest post by Kazuma Nagano (CyberAgent).

Guest blog post by Kazuma Nagano. The original article was posted in Japanese.

Kazuma Nagano is an iOS engineer at CyberAgent who works in the iOS team of Tapple (Japanese dating apps).

Workflow before and after tuning

My team is now automating tests on Bitrise and the workflow is as follows.

- Clone repository
- Install Homebrew package (Xcodegen, Linter, etc.)
- Gem package installation (xcov, CocoaPods, danger, fastlane, etc.)
- Install Cocoapods
- Run tests

The following is an actual summary of the workflow before acceleration (with the last execution time displayed).

Before tuning


+------------------------------------------------------------------------------+
|                               bitrise summary                                |
+---+---------------------------------------------------------------+----------+
|   | title                                                         | time (s) |
+---+---------------------------------------------------------------+----------+
| ✓ | activate-ssh-key                                              | 11.26 sec|
+---+---------------------------------------------------------------+----------+
| ✓ | git-clone                                                     | 1.5 min  |
+---+---------------------------------------------------------------+----------+
| ✓ | Set Env Path                                                  | 5.94 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | cache-pull                                                    | 4.0 min  |
+---+---------------------------------------------------------------+----------+
| ✓ | Bundle Install                                                | 50.90 sec|
+---+---------------------------------------------------------------+----------+
| ✓ | Brew Install                                                  | 1.1 min  |
+---+---------------------------------------------------------------+----------+
| ✓ | XcodeGen                                                      | 15.0  sec|
+---+---------------------------------------------------------------+----------+
| ✓ | CocoaPods Install                                             | 4.1 min  |
+---+---------------------------------------------------------------+----------+
| ✓ | cache-push                                                    | 25.18 sec|
+---+---------------------------------------------------------------+----------+
| ✓ | github-status                                                 | 5.69 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | Fastlane Test                                                 | 19.0 min |
+---+---------------------------------------------------------------+----------+
| ✓ | cache-push                                                    | 15.52 sec|
+---+---------------------------------------------------------------+----------+
| ✓ | github-status                                                 | 5.19 sec |
+---+---------------------------------------------------------------+----------+
| Total runtime: 31.9 min                                                      |
+------------------------------------------------------------------------------+
Copy code

Especially in this, we reviewed each setting of Homebrew, Gem, Cocoapods, (App build). By making better use of the cache, we were able to speed up the total workflow time by 40-50% as shown below.

After tuning



+------------------------------------------------------------------------------+
|                               bitrise summary                                |
+---+---------------------------------------------------------------+----------+
|   | title                                                         | time (s) |
+---+---------------------------------------------------------------+----------+
| ✓ | activate-ssh-key                                              | 10.15 sec|
+---+---------------------------------------------------------------+----------+
| ✓ | Git Config                                                    | 3.17 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | git-clone                                                     | 29.22 sec|
+---+---------------------------------------------------------------+----------+
| ✓ | Set Env Path                                                  | 4.44 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | cache-pull                                                    | 31.66 sec|
+---+---------------------------------------------------------------+----------+
| ✓ | Brew install                                                  | 8.13 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | XcodeGen                                                      | 11.53 sec|
+---+---------------------------------------------------------------+----------+
| ✓ | rbenv Install                                                 | 4.48 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | Bundle Install                                                | 5.07 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | CocoaPods Install                                             | 38.11 sec|
+---+---------------------------------------------------------------+----------+
| ✓ | cache-push                                                    | 9.71 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | github-status                                                 | 6.70 sec |             
+---+---------------------------------------------------------------+----------+
| ✓ | Fastlane Test                                                 | 13.6 min |
+---+---------------------------------------------------------------+----------+
| ✓ | github-status                                                 | 7.55 sec |
+---+---------------------------------------------------------------+----------+
| Total runtime: 16.4 min                                                      |
+------------------------------------------------------------------------------+
Copy code

The total runtime on the Bitrise summary varies depending on the time zone and environment. The summary logs we are comparing in this article are compared at the same time, but other factors may also have an impact.

Appendix

In the pre-tuning workflow above,

  1. . / Pod (if Podfile.lock has changed)
  2. / Users / vagrant / Library / Caches / Homebrew
  3. ~ / Library / Developer / Xcode / DerivedData

was cached.

In the workflow after acceleration, steps 2 and 3 were deleted because they were not effective.

Cache function in Bitrise

Bitrise docs have a description of the cache functionality.

Some excerpts

  • Cache tars all cached directories and dependencies and stores them securely in Amazon S3
  • If a / path / to / cache is specified,/ path / to / cache / .ignore-me is also cached
  • Cache is managed in branch units
  • Cache-push is not performed in PRs
  • If no new build is done on that branch, it will expire and be deleted after 7 days

Use

It can be used by adding two steps to the workflow.

Bitrise.io Cache:Pull step to download the previous cache (if any).
Bitrise.io Cache:Push step to check the state of the cache and upload it if required.

By writing the following in the Bitrise workflow, you can cache under the selected directory.

bitrise.yml


workflows:
    workflow-a:
        steps:
        # - Others
        - cache-pull: {} #Only this line for pull
        # - Others
        - cache-push:
        inputs:
        - cache_paths: |-
            $BITRISE_CACHE_DIR
       MyProject/Cache  #Cache target path
            ./Pods -> ./Podfile.lock #If Podfile.lock changes, cache ./Pods
Copy code

You can specify a file to watch for changes like the example in Podfile.lock. But it doesn't mean that the system doesn't check diff in other cases.

Log when caching is successful


Cleaning paths
Done in 1.32941661s
Checking previous cache status
Previous cache info found at: /tmp/cache-info.json
Done in 171.741223ms
Checking for file changes
0 files needs to be removed
0 files has changed
0 files added
No files found in 37.785423ms
Total time: 1.539638885s
|                                                                              |
+---+---------------------------------------------------------------+----------+
| ✓ | cache-push                                                    | 8.99 sec |
+---+---------------------------------------------------------------+----------+
Copy code

If you check the logs, you can see that each deletion, modification, and addition is confirmed. So you can simplify this flow by specifying a file to monitor for changes.

Cache settings that actually worked

In this chapter, we introduce a caching method that we have tried and found effective.

Homebrew cache

In the Bitrise Docs - Caching Homebrew installers, the following methods are described,

The Brew install Step supports caching: if the option is enabled, any downloaded brew installers will be cached from the location of brew --cache. The cache path is ~/Library/Caches/Homebrew/.
To enable caching of brew installers:
  • Go to the Workflow in which you want to cache brew installs and select the Brew install Step.
  • Set the Cache option to yes.
  • As always, click Save.

In this description, Homebrew’s --cache option is specified. This method, however, runs the installation every time, which - though not necessary - takes extra time.

By using the method of directly linking binaries, you can avoid re-installing the installed fomula.
(I recommend that you don’t use the official Brew Install Step because the cache in ~/Library/Caches/Homebrew has little time-saving contribution)

bitrise.yml


- script:
        inputs:
        - content: |-
            # Add cache directory to environment variable
            envman add --key BREW_XCODEGEN --value "$(brew --cellar)/xcodegen" #Specify youre package name
            envman add --key BREW_OPT_XCODEGEN --value "/usr/local/opt/xcodegen"  #Specify youre package name
        title: Set Env Path
- script:
        inputs:
        - content: |-
            # Install the following command
            brew install xcodegen
            brew link xcodegen
        title: Brew install
- cache-push:
        inputs:
        - cache_paths: |-
           $BITRISE_CACHE_DIR
           # Add the following two to cache_path
           $BREW_XCODEGEN
           $BREW_OPT_XCODEGEN
Copy code

Log when the cache is successful


+ brew install xcodegen
Warning: xcodegen 2.10.1 is already installed, it's just not linked
You can use `brew link xcodegen` to link this version.
+ brew link xcodegen
Linking /usr/local/Cellar/xcodegen/2.10.1... 2 symlinks created
Copy code

Before tuning


+---+---------------------------------------------------------------+----------+
| ✓ | Brew Install                                                  | 1.1 min  |
+---+---------------------------------------------------------------+----------+
Copy code


After tuning


+---+---------------------------------------------------------------+----------+
| ✓ | Brew install                                                  | 8.13 sec |
+---+---------------------------------------------------------------+----------+
Copy code

Also, with this method, if one or more installations are run, brew cleanup will run on the same folders every time.

undefined

You can check the condition in cleanup.rb in the Homebrew Github repository.
The forced cleanup process is determined using ~ / Library / Caches / Homebrew / .cleaned, so it can be avoided by adding this to the cache target.

Ruby Gems

rbenv

In Bitrise Docs - Caching Ruby Gems, The following methods are described.

undefined

In my local environment, this cache path refers to ~ / .rbenv / versions / 2.6.3. This is the cache specification for rbenv.
In the stack provided by Bitrise, ruby 2.6.3 is not installed, so this setting was effective.

Also, if you manage .ruby-version in the repository, check this when you push the cache.

gem

In addition, by caching the gem installation directory (. / Vendor / bundler on Tapple)
You can also avoid installing during bundle install.
Since this is reflected in Gemfile.lock, specify the check at the time of push.

bitrise.yml


- script:
        title: Bundle Install
        inputs:
        - content: |-
            bundle install --path vendor/bundle
- cache-push:
        inputs:
        - cache_paths: |-
            #Add the following two to cache_path
            $ GEM_CACHE_PATH-> ./.ruby-version #Environment variables in the official document ↑
            ./vendor-> ./Gemfile.lock #gem installation directory
            

Copy code

Log when the cache is successful


+ bundle install --path vendor/bundle
Using rake 13.0.1
Using CFPropertyList 3.0.1
...
Using xcprofiler 0.6.3
Bundle complete! 9 Gemfile dependencies, 109 gems now installed.
Bundled gems are installed into `./vendor/bundle`
Copy code

This reduced the time of bundle install by 90%.

Before Tuning


+---+---------------------------------------------------------------+----------+
| ✓ | Bundle Install                                                | 50.90 sec|
+---+---------------------------------------------------------------+----------+
Copy code

After tuning


+---+---------------------------------------------------------------+----------+
| ✓ | Bundle Install                                                | 5.07 sec |
+---+---------------------------------------------------------------+----------+
Copy code


CocoaPods

In, Bitrise Docs - Caching Ruby Gems, The following method is described.

Before you start, make sure you have the latest version of the Cocoapods Install Step in your Workflow.
  • Open your app’s Workflow Editor.
  • Insert the Cache:Pull Step after the Git Clone but before the Cocoapods Install steps.
    IMPORTANT: Make sure that your Step is version 1.0.0 or newer. With the older versions, you have to manually specify paths for caching.
  • Insert the Cache:Push step to the very end of your workflow.

It seems that it will work just by using the official Bitrise Cocoapods Install step.

But, in my case, I checked the log during the execution of the official step,

  • Check if selected Ruby is installed
  • $ gem install bundler --force
  • bundle install
  • pod install

It does a lot of work, but there is an overlap with the previous step.

undefined

So I run the bundle exec pod install script alone.

If the correct bundle and rbenv is installed before the step,
You can shorten it by simply specifying Pods file as the cache path and checking Podfile.lock.

bitrise.yml


- script:
        inputs:
        - content: |-
            bundle exec pod install
        title: CocoaPods Install
- cache-push:
        inputs:
        - cache_paths: |-
        ./Pods -> ./Podfile.lock ##キャッシュパスに追加
Copy code

In the pre-tuning workflow, the --repo-update option was specified every time, but it only has to be run if there is an update.
You can also keep a valid CocoaPods cache.
As a result, we were able to save about 85%.

Before tuning


+---+---------------------------------------------------------------+----------+
| ✓ | CocoaPods Install                                             | 4.1 min  |
+---+---------------------------------------------------------------+----------+
Copy code

After tuning


+---+---------------------------------------------------------------+----------+
| ✓ | CocoaPods Install                                             | 38.11 sec|
+---+---------------------------------------------------------------+----------+
Copy code


Log when the cache is successful


+ bundle exec pod install
Using FBSDKCoreKit
Using FirebaseDynamicLinks
Using Crashlytics
Using FirebaseCoreDiagnosticsInterop
Using FirebaseCore
Using GoogleDataTransport
Using GoogleDataTransportCCTSupport
・・・
Copy code

For further acceleration

When you look at the pre-tuning summary, you can see that the Fastlane Test runs the longest.


+---+---------------------------------------------------------------+----------+
| ✓ | Fastlane Test                                                 | 19.0 min |
+---+---------------------------------------------------------------+----------+
Copy code

In this step

  • Build app
  • Test
  • Danger
  • Notification to Slack

are included.

Is there any way to speed up this part using a cache?

cocoapods-binary

Unfortunately, Bitrise currently doesn’t support Xcode build cache, but CocoaPods’s _Prebuild using cocoapods-binary can be cached.

https://github.com/leavez/cocoapods-binary (The introduction is omitted in this article.)

By caching . / Pods / _Prebuild (because it is under Pods, you do not need to specify an additional cache path), you do not need to build Pods every time.
The build time on CI was greatly reduced: this was a 30% improvement, but this was the biggest time we were able to achieve (5.4m).

Before tuning


+---+---------------------------------------------------------------+----------+
| ✓ | Fastlane Test                                                 | 19.0 min |
+---+---------------------------------------------------------------+----------+
Copy code

After tuning


+---+---------------------------------------------------------------+----------+
| ✓ | Fastlane Test                                                 | 13.6 min |
+---+---------------------------------------------------------------+----------+
Copy code


Supplement

In this article, we introduced the segments of a build that can make use of Bitrise’s cache the most.
But don’t forget that storing and using the cache always has the risk of inconsistencies somewhere.

If you look at the Bitrise cache log, you’ll see that it confirms that the stack is the same before pulling.

cache-pull logs


+------------------------------------------------------------------------------+
| (4) cache-pull                                                               |
+------------------------------------------------------------------------------+
| id: cache-pull                                                               |
| version: 2.1.2                                                               |
| collection: https://github.com/bitrise-io/bitrise-steplib.git                |
| toolkit: go                                                                  |
| time: 2019-12-02T15:03:01Z                                                   |
+------------------------------------------------------------------------------+
|                                                                              |
Config:
- CacheAPIURL: [REDACTED]
- DebugMode: false
- StackID: osx-xcode-11.2.x
Downloading remote cache archive
Checking archive and current stacks
current stack id: osx-xcode-11.2.x
archive stack id: osx-xcode-11.2.x
Extracting cache archive
Done
Copy code


Measures such as

  • Set TTL such as *Discard cache on library update
  • Prepare a way to refresh and invalidate the cache at any time are required.

With this implementation, I felt that it was important to build relationships that depended on the cache, but that did not depend too much.

References

No items found.

Explore more topics

App development

Best practices from engineers on how to use Bitrise to build better apps, faster.

Community

Meet other Bitrise engineers, technology experts, power users, partners and join our BUGs.

Company

All the updates about Bitrise events, sponsorships, employees, and more.

Insights

Mobile development, latest tech, industry insights, and interviews with experts.

Mobile DevOps

Learn why mobile development is unique and requires a set of unique practices.

Releases

Stay tuned for the last updates, new features, and product improvements.

Get the latest from Bitrise

Join other Mobile DevOps engineers who receive regular emails from Bitrise, filled with tips, news, and best practices.