Working with Fastlane on iOS in an Enterprise Environment

In case you were wondering, no — Fastlane doesn’t have an enterprise offering. It’s always been (and I hope will continue to be) an open-source product.

Fastlane is a collection of tools that help you automate all aspects of mobile development, such as creating new app IDs in the developer portal and managing provisioning profiles; building, testing and packaging apps; and distributing apps to the App Store.

What this article really centers on is how to use Fastlane while working in an enterprise environment. Specifically those Fastlane actions that need a connection to external URLs.

Enterprise Environment

If you’re one of the lucky devs who can just “connect to Internet directly”™️, you may as well just skip this article and get back to enjoying Getting Things Done.

If, my unlucky friend, phrases like company proxy and company firewall sound familiar, then you may find the rest of the article useful.

So, in very simple terms for non-DevOps engineers like me, working in a big enterprise company usually means that there’s at least one proxy server sitting between you—the developer—and the outside world, aka The Internet. The proxy will make sure to authenticate you securely, just to know who you are; and it will do other important things, like peek inside every single zip archive you download. Why? Because it can…

To make sure you don’t visit harmful websites like GitHub Gists (🤯), there’s also a network firewall thrown into the mix. The firewall will only allow connections to the whitelisted endpoints.

Whitelist

If you’re the one tasked with a job of “Uploading ipa file to TestFlight from CI server”, the first thing to do is to request access to Apple endpoints via your company’s proxy and firewall. Mind you, the journey you’re about to embark on will make Homer’s Odyssey look like a pleasant weekend trip.

Depending on the complexity of your company’s IT infrastructure and level of support, you’d better collect all the information on your side first (and check it twice) before submitting any requests.

Now, Spaceship is the core component of Fastlane that enables access both to Apple Developer Center and to App Store Connect. The list of API endpoints used by Spaceship is a good starting point for the request you’re about to submit.

I’d even recommend going one step further and requesting whitelist access for all URLs on theapple.com domain. It will make your setup more future-proof, in case Apple adds new APIs or changes existing private APIs, which they do quite often.

The final list of domains to whitelist should be like this:

  • *.apple.com/* – for all kinds of Apple stuff.
  • *.devimages.apple.com.edgekey.net/* – this is not strictly required for communicating with the dev portal or App Store Connect, but this is where Fastlane can download iOS Simulator images.
  • *.mzstatic.com/* – so that Fastlane can download screenshots of your app from App Store Connect.

The port to request access is 443 – for all HTTPS traffic.

🤖 Service Account

Before you go ahead with submitting a new request for your IT infrastructure change, make sure you have a service account as well.

Unlike normal user accounts, service accounts are not tied to a particular employee. This is very important. When a person leaves a company, their account eventually gets wiped and CI pipelines get broken, so the lesson is: never use personal accounts for CI setup. Service accounts don’t ask for a pay raise, don’t chuck a sickie, and they just don’t quit.

Another good thing about service accounts—they normally don’t have the password expiry policy set. You don’t have to update your CI configuration with a new password every 90 days or so.

If you don’t have a service account, request one then. Yes, it may mean you have to open yet another request to some other DevOps team, but who said enterprise was easy?

Using Fastlane behind the Proxy

Fast-forward N weeks, where N is a random large non-negative number, and you have the following at your disposal:

  • A service account username and password, and maybe even an email, if you’re super lucky.
  • Let’s assume the username is part of the Active Directory Domain and comes in a auusername form, where au is the domain.
  • Let’s also assume the password is just password.
  • All requested domains are whitelisted on all levels of intranet.
  • You service account can authenticate to the company proxy using username and password and access whitelisted domains.
  • Let’s assume the proxy host name is company-proxy and it listens on port 8080 for all incoming connections.

Curl Test

First off, run a simple curl test to make sure everything works:

curl 
    --proxy-user au\username:password 
    --proxy http://company-proxy:8080 
    --proxy-anyauth 
    --insecure 
    --verbose 
    "https://appstoreconnect.apple.com"

The –proxy-user and –proxy options are self-explanatory, though note the backslash escaping as \.

By saying –proxy-anyauth we tell curl to work with any authentication scheme that the proxy supports.

Most likely, all outgoing network traffic on your dev machine is signed with your company’s own self-signed Root Certificate Authority (CA) certificate, so we use the –insecure option to convince curl to accept this self-signed certificate.

The self-signed root CA and authentication scheme will play very important roles further on.

The output of the command should be something like this:

This is good. It means the request went all the way to appstoreconnect.apple.com.

🔏 Root CA and SSL Certificates

Just like with curl, a self-signed root CA certificate won’t be accepted by Fastlane either. While running Fastlane, you won’t have any option like the curl’s –insecure flag. Instead you need to add your internal root CA certificate to rubygems path, because Fastlane is a Ruby gem:

# Name of your internal root Certificate Authority.
ROOT_CA_NAME=MyCompanyRootCA

# Get rubygems parentdirectory.
RUBYGEMS_PATH=$(dirname $(gem which rubygems))

# Get path to SSL certificates storage.
CERTS_DIR=${RUBYGEMS_PATH}/rubygems/ssl_certs/rubygems.org
mkdir -p ${CERTS_DIR}

# Find the root CA cert in keychain.
INTERNAL_ROOT_CA=$(security find-certificate -c ${ROOT_CA_NAME} -p)

# Write the root CA cert to rubygems location.
echo "${INTERNAL_ROOT_CA}" >"${CERT_PATH}/${ROOT_CA_NAME}.pem"

You may have to use sudo if you’re using the system Ruby version, though I’d recommend using RVM or rbenv to manage your rubies.

If you are using RVM, you’ll need to run the following as well:

rvm osx-ssl-certs update all --force

Now you can use this Ruby script to test the SSL connection.

E.g. just run it directly like this:

ruby <(curl -s -L https://gist.github.com/mgrebenets/ebab9213959319779807ce39bd0bed56/raw)

💲 Environment Variables

The next thing you’ll notice is that Fastlane doesn’t have any option for passing proxy information. Luckily, Fastlane is being a good citizen and respects all the /http[s]_proxy/i environment variables, namely:

  • HTTP_PROXY
  • HTTPS_PROXY
  • http_proxy
  • https_proxy

Remember this “Gang of Four”—nothing else will cause you as much pain as this bunch!

Somewhere in your ~/.bash_profile or other appropriate shell profile, set these environment variables—for example:

COMMON_PROXY=http://au\username:password@company-proxy:8080

export HTTP_PROXY=$COMMON_PROXY
export HTTPS_PROXY=$COMMON_PROXY
export http_proxy=$COMMON_PROXY
export https_proxy=$COMMON_PROXY

Note that proxy URL is set in the following format:

Without the username and password, you wouldn’t be able to authenticate with your proxy.

Now give it a try. Run the simple download_screenshots deliver action:

bundle exec fastlane deliver download_screenshots

Answer all the prompts and see if it worked. I bet it didn’t!

Long story short, Fastlane is using the faraday HTTP client Ruby library, which uses Kernel.URI method, which can’t handle a backslash in the username.

The solution is to use percent-encoding for , i.e. %5c:

BACKSLASH=%5c
COMMON_PROXY=http://au${BACKSLASH}username:password@company-proxy:8080

You should be able to download screenshots now, though that’s just a first, tiny step towards the final goal. Well, actually, you can do a lot of things now, such as registering devices, managing App IDs and provisioning profiles, and more. But I’d assume you also want to upload a new ipa to TestFlight, so meet The Transporter then…

🚎 Transporter

Transporter, aka iTMSTransporter, is a Java-based command-line tool used to “upload things to the App Store”, including app metadata and new ipa builds.

Installing Transporter

So how do you get a copy?

Well…according to Apple, you first login to your developer account and then click the download link like this one.

But wait! You can’t download it unless you have an Admin or Technical role.

I mean, “Why, Apple?! Why?!” 😱

Transporter is available as part of the Xcode installation anyways…why make things so hard?

Why not make it available at Apple Developers Downloads at the very least?

Anyways, if you choose to use the copy bundled with Xcode, then you can find it at:

Just grab the whole directory.

If you installed it via the Mac OS package, then get it at:

Patching Transporter

If you’re wondering: “Why do I need a copy of Transporter? Can’t Fastlane just use it as is?”

Good question. You don’t need a copy of Transporter. You can use the one available inside Xcode or in your /usr/local, but you would have to patch it before using it anyways.

Given that Transporter is a Java-based command-line app, it has the same issue as curl or any Ruby gem would have with your company’s internal self-signed root CA certificate – it won’t trust it. Apple didn’t bother to add a command line or configuration option of any kind to Transporter to trust self-signed certificates.

Thankfully, Transporter isn’t just a Java-based app, it comes with its own version of Java runtime bundled up inside itms/java. Try running itms/java/bin/java -version to see more info.

Now, the itms/java folder also contains lib/security/cacerts file, which is a keystore with CA certificates. So you need to add your company certificate to cacerts, and itms/java/bin/keytool is just the right tool to do that.

# Run this inside Transporter's itms directory.
# Patch the trasporter with internal root CA.
./java/bin/keytool -import -trustcacerts 
    -alias internal-root-ca 
    -file ${ROOT_CA_PATH} 
    -keystore java/lib/security/cacerts 
    -storepass changeit 
    -noprompt 
    -v

Here ${ROOT_CA_PATH} is the path to the certificate file, which you can locate directly or find in the OS X keychain

ROOT_CA_NAME=MyCompanyRootCA
ROOT_CA_PATH=root-cert.txt
$(security find-certificate -c ${ROOT_CA_NAME} -p) > ${ROOT_CA_PATH}

Now you’ve convinced Transporter to trust your root CA, but you’re not there yet, because you also need to teach Transporter some respect towards your new and shiny proxy, which is where trouble awaits for you…again.

Transporter and Proxy

Now that we know that Transporter is a Java-based app, we look closer inside theitms folder and find itms/java/lib/net.properties – a Java properties file used to configure the JVM that Transporter runs in.

The group of proxy variables is exactly what we need:

# net.properties
http.proxyHost=company-proxy
http.proxyPort=8080
https.proxyHost=company-proxy
https.proxyPort=8080

You don’t even have to modify the net.properties file. Fastlane reads the value of the special DELIVER_ITMSTRANSPORTER_ADDITIONAL_UPLOAD_PARAMETERS environment variable and uses it to configure Transporter. For example, in your bash script add the following export statement:

# Set environment variable with proxy configuration for Transporter.
PROXY_HOST=company-proxy
PROXY_PORT=8080
PARAMS="-Dhttp.proxyHost=${PROXY_HOST}"
PARAMS+=" -Dhttp.proxyPort=${PROXY_PORT}"
PARAMS+=" -Dhttps.proxyHost=${PROXY_HOST}"
PARAMS+=" -Dhttps.proxyPort=${PROXY_PORT}"

export DELIVER_ITMSTRANSPORTER_ADDITIONAL_UPLOAD_PARAMETERS="${PARAMS}"

# Run Fastlane next.

But will it work? By now you’ve been there enough times to know the answer — “No”. 😜

Proxy configuration needs to include the username and password for Basic authentication.

You can try to add au%5cusername:password@company-proxy as the host value, but it won’t work.

Removing the jdk.http.auth.tunneling.disabledSchemes=Basic limitation from net.properties won’t help either.

While trying to Google you may find mention of these properties:

http.proxyUser
http.proxyPassword
https.proxyUser
https.proxyPassword

Those are part of Apache HTTP Client JVM options, though, and are not part of Oracle’s default network properties. Also, Transporter doesn’t respect them.

🚧 Feels like another roadblock.

Another Proxy

The solution this time is to create yet another proxy (as if there weren’t enough problems with them already!).

This “man in the middle” local proxy will take care of passing authentication information to your company proxy. The credentials no longer have to be hard-coded in a proxy URL. This way, the command line tools like Transporter can use a local proxy for network connections, and a local proxy doesn’t require any type of authentication.

There are a number of proxy tools available for Mac OS. In this example, I’ll be using CNTLM.

I’ll briefly mention that I also tried TinyProxy and ProxyChains-NG. TinyProxy just wouldn’t let me use or %5c in the configuration file, while with ProxyChains-NG I eventually just gave up after a number of failed attempts to make it work.

So, CNTLM—a fast proxy for NTLM authentication implemented in C.

Easy to install.

brew install cntlm

Also easy to configure. Just put the following in cntlm.conf:

Domain      au
Username    username
Proxy       company-proxy:8080
NoProxy     *.local, 169.254/16
Listen      9128

Now run this command to generate password hashes:

cntlm -c cntlm.conf -H

Type in your account password, copy hashes, and append them to cntlm.conf:

PassLM          1AD35398BE6565DDB5C4EF70C0593492
PassNT          77B9081511704EE852F94227CF48A793
PassNTLMv2      7692C44DD39AFB2FBB026DA48A7B6FE2    # Only for user 'username', domain 'au'

Then run it:

# Use -f to run in foreground mode.
cntlm -c cntlm.conf -f

# Run in background, e.g. if it's part of CI shell script.
cntlm -c cntlm.conf&

Finally, configure proxy environment variables:

# No more authentication!
PROXY_HOST=localhost
PROXY_PORT=9128
COMMON_PROXY=http://${PROXY_HOST}:${PROXY_PORT}

export HTTP_PROXY=$COMMON_PROXY
export HTTPS_PROXY=$COMMON_PROXY
export http_proxy=$COMMON_PROXY
export https_proxy=$COMMON_PROXY

PARAMS="-Dhttp.proxyHost=${PROXY_HOST}"
PARAMS+=" -Dhttp.proxyPort=${PROXY_PORT}"
PARAMS+=" -Dhttps.proxyHost=${PROXY_HOST}"
PARAMS+=" -Dhttps.proxyPort=${PROXY_PORT}"

export DELIVER_ITMSTRANSPORTER_ADDITIONAL_UPLOAD_PARAMETERS="${PARAMS}"

Give it a go!

curl 
    --proxy http://localhost:9128 
    --proxy-anyauth 
    --insecure 
    --verbose 
    "https://appstoreconnect.apple.com"

What do you mean “It doesn’t work!?” Something about the proxy returning an invalid challenge?

Well, that just means you company proxy doesn’t support NTLM authentication and only supports Basic authentication. That needs to be changed by requesting IT support again, I’m afraid.

Now the curl test should finally work, and your configuration now looks like this:

Upload ipa

You’re all set now to run one of the most common and recurring tasks in the Continuous Delivery process for an iOS app — upload a new build to TestFlight.

Start by setting up the basic lane using pilot action.

# Fastfile.

lane :test_ipa_upload do
  pilot(
    username: "[email protected]",
    apple_id: "app Adam/Apple/App ID",
    ipa: "path/to/ipa",
    app_identifier: "bundle.id",
    skip_submission: true,
    skip_waiting_for_build_processing: true
  )
end

To skip the password prompt, set the FASTLANE_PASSWORD environment variable. It could be a secret parameter in your CI server configuration, or you could save it in your OS X keychain as AppleID.Password and read like so:

export FASTLANE_PASSWORD="$(security find-generic-password -a [email protected] -s AppleID.Password -w)"

It’s important not to confuse the Apple ID used as username with the app’s Apple ID. The latter is the unique identifier the app is assigned in the App Store—that’s why it’s often referred to as Adam or Adam/Apple/App ID. You can find out the Adam ID of the app by navigating to it in App Store Connect and grabbing the last part of the URL.

Then again, the app_identifier parameter is actually the Bundle Identifier stored under CFBundleIdentifier in the application’s Info.plist.

In this example, we don’t want to submit a new build for review yet, and we don’t want to wait for processing either, thus both skip_submission and skip_waiting_for_build_processing are set to true.

Note that with most of Fastlane actions, each action parameter is shadowed by an environment variable. For example, instead of setting the ipa parameter, you can set the PILOT_IPA environment variable. To see a full list of parameters for an action, run bundle exec fastlane action <action_name>, for example:

bundle exec fastlane action pilot

Try to run the lane now—after setting up all the environment variables, of course:

bundle exec fastlane test_ipa_upload

🤝 Fastlane Session

If you plan to use similar action on CI server, then Two-factor authentication (2FA) enforced by Apple will become an issue. To avoid 2FA on CI servers, you need to set the FASTLANE_SESSION environment variable as described here.

bundle exec fastlane spaceauth -u [email protected]

Whenever using 2FA, the browser will prompt you, asking you if you want to trust this particular browser. Doing so will generate a long-living session token (30 days), which will then be picked by Fastlane’s spaceauth command. This way you’ll only have to update your CI server configuration once every 30 days. There may be some ways to keep the session token up to date by running scheduled CI jobs that perform Spaceship login, though I haven’t tested that approach yet.

Summary

By the end of this journey, you should be able to upload new iOS app builds to TestFlight from the enterprise network, e.g. from a CI build agent.

It’s amazing how such a basic task requires so much extra effort when dealing with all the overhead inherent to most large companies.

I hope it was worth the effort, though. Once it’s all set up and runs smoothly, there are so many other aspects of iOS development that can be automated using tools like Fastlane.

Avatar photo

Fritz

Our team has been at the forefront of Artificial Intelligence and Machine Learning research for more than 15 years and we're using our collective intelligence to help others learn, understand and grow using these new technologies in ethical and sustainable ways.

Comments 0 Responses

Leave a Reply

Your email address will not be published. Required fields are marked *