Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document Flutter ios testing support #3066

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 150 additions & 0 deletions docs/dev/api/rdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -1014,6 +1014,156 @@ curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" --location \

---

### Start a XCTest, XCUITest or Espresso Job

<details>
<summary><span className="api post">POST</span> <code>/v1/rdc/native-composer/tests</code></summary>
<p/>

Start a XCTest.

#### Parameters

<table id="table-api">
<tbody>
<tr>
<td><code>test_framework</code></td>
<td><p><small>| BODY | REQUIRED | STRING |</small></p><p>The type of 'native' test you want to run, either 'XCUITEST', 'XCTEST', or 'ANDROID_INSTRUMENTATION' (Espresso).</p></td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>app_id</code></td>
<td><p><small>| BODY | REQUIRED | STRING |</small></p><p>The id of the app you want to test. The app must be uploaded to Sauce Labs AppStorage. Format: `storage://filename=AnApp.ipa`, or `storage://{file_id}`, or just `{file_id}`.</p></td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>xc_test_run_file</code></td>
<td><p><small>| BODY | REQUIRED | STRING |</small></p><p>The id of the `xctestrun` file (the xctest config). The file must be uploaded to Sauce Labs AppStorage. Format: `storage://filename=Runner.xctestrun`, or `storage://{file_id}`, or just `{file_id}`.</p></td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>device_query</code></td>
<td><p><small>| BODY | REQUIRED | OBJECT |</small></p><p>Defines on which device you want to run you test. The available attributes are:<ul><li><code>type</code> - <small>String of 'DynamicDeviceQuery'</small></li><li><code>device_name</code> - <small>String - regular expression to select a device.</small></li><li><code>os_version</code> - <small>String - regular expression to select a device version.</small></li></ul></p><p>The <code>type</code> parameter is required..</p></td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>tunnel_name</code></td>
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Assign a sauce connect tunnel to the job.</p></td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>tunnel_owner</code></td>
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Define the name of the tunnel owner of the sauce connect tunnel you want to use.</p></td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>settings_overwrite</code></td>
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Define instrumentations settings you want to apply to your app.</p></td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>test_name</code></td>
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Assign a name to the job, to be displayed in the UI.</p></td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>build</code></td>
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Assign the job to a build. You can specify an existing build name or create a new one.</p></td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>tags</code></td>
<td><p><small>| BODY | OPTIONAL | ARRAY |</small></p><p>The set of tags to apply to the job.</p></td>
</tr>
</tbody>
</table>

<Tabs
groupId="dc-url"
defaultValue="us"
values={[
{label: 'United States', value: 'us'},
{label: 'Europe', value: 'eu'},
]}>

<TabItem value="us">

```jsx title="Sample Request"
curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" --location \
--request POST 'https://api.us-west-1.saucelabs.com/v1/rdc/native-composer/tests' \
--header 'Content-Type: application/json'
--data-raw '{
"test_framework": "XCTEST",
"test_name": "Your XCTest POC",
"app_id": "9349d683-5d26-46eb-a943-267cbe8b4deb",
"xc_test_run_file": "37eef454-a7f8-4a6d-a340-d39865ad6db3",
"device_query": {
"type": "DynamicDeviceQuery"
}
}'
```

</TabItem>

<TabItem value="eu">

```jsx title="Sample Request"
curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" --location \
--request POST 'https://api.eu-central-1.saucelabs.com/v1/rdc/native-composer/tests' \
--header 'Content-Type: application/json'
--data-raw '{
"test_framework": "XCTEST",
"test_name": "Your XCTest POC",
"app_id": "9349d683-5d26-46eb-a943-267cbe8b4deb",
"xc_test_run_file": "37eef454-a7f8-4a6d-a340-d39865ad6db3",
"device_query": {
"type": "DynamicDeviceQuery"
}
}'
```

</TabItem>
</Tabs>

#### Responses

<table id="table-api">
<tbody>
<tr>
<td><code>200</code></td>
<td colSpan='2'>Success.</td>
</tr>
</tbody>
<tbody>
<tr>
<td><code>400</code></td>
<td colSpan='2'>Bad Request.</td>
</tr>
</tbody>
</table>

```jsx title="Sample Response"
{
"test_report": {
"id": "ef6058735d8d4dfa9e0077d250757aac",
"url": "https://api.eu-central-1.saucelabs.com/tests/ef6058735d8d4dfa9e0077d250757aac"
}
}
```

</details>

---

### Stop a Job

<details>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Follow this guide to run [integration tests](https://docs.flutter.dev/cookbook/t
5. Create a directory called `integration_test` in the root of your Flutter project.
6. Create a file called `flutter_integration_test.dart` in the `integration_test` directory.
7. Update your testing dart file `flutter_integration_test.dart` to include the ***tearDownAll***,
The purpose for this is to make sure we close the connection to the driver after the tests have completed.
The purpose for this is to make sure we close the connection to the device after the tests have completed.
```dart
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
Expand Down Expand Up @@ -118,7 +118,7 @@ Follow this guide to run [integration tests](https://docs.flutter.dev/cookbook/t

9. Configure `saucectl` to run the test.
* Create a folder `saucectl` in your project root directory.
* Inside this folder create a `flutter_integration_test.yaml` with the following content:
* Inside this folder create a `flutter_integration_test_android.yaml` with the following content:
```yaml
apiVersion: v1alpha
kind: espresso
Expand Down Expand Up @@ -207,12 +207,4 @@ Follow this guide to run [integration tests](https://docs.flutter.dev/cookbook/t
## Example Implementation

For a practical example of how to set up and run integration tests for Flutter apps, you can refer to the [Sauce Labs Flutter demo application](https://github.com/saucelabs/my-demo-app-flutter) repository.
The steps outlined in this guide have already been implemented in that repository. You can follow along with the demo app to see how everything is configured and run your tests accordingly.

## What's Next

:::info Next step

We're excited to share that Sauce Labs is actively working on expanding support for Flutter integration tests on iOS.
Stay tuned for updates as we continue to develop this capability!
:::
The steps outlined in this guide have already been implemented in that repository. You can follow along with the demo app to see how everything is configured and run your tests accordingly.
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
---
id: flutter-integration-testing-ios
title: Flutter iOS
sidebar_label: Flutter iOS
description: Run your Flutter integration tests for iOS
---

import useBaseUrl from '@docusaurus/useBaseUrl';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

Flutter compiles iOS [integration tests](https://docs.flutter.dev/cookbook/testing/integration/introduction) into [XCTests](https://developer.apple.com/documentation/xctest) so that they can be executed on Apple devices. The following will explain how to run your XCTests on Sauce Labs infrastructure.

**To run an XCTest (or 'Flutter test') on Sauce Labs, you need to provide two test artifacts:**
1. Your flutter-iOS app compiled as an `.app` or `.ipa` file.
2. The `.xctestrun` file for that app. The [.xctestrun file](https://keith.github.io/xcode-man-pages/xcodebuild.xctestrun.5.html) is the config for your test, this is the same config that Xcode uses when it runs your tests on your development machine.


## Contents
1. [How to build the '.app' and '.xctestrun' files for your Flutter app.](#1-how-to-build-the-app-and-xctestrun-files-for-your-flutter-app)
2. [How to run the flutter-iOS integration test on Sauce Labs infrastructure.](#2-how-to-run-flutter-ios-integration-tests-on-sauce-labs-infrastructure)
3. [Sample Implementation](#example-implementation)


:::info What You'll Need

- A Sauce Labs account ([Log in](https://accounts.saucelabs.com/am/XUI/#login/) or sign up for
a [free trial license](https://saucelabs.com/sign-up))
- Your Sauce Labs [Username and Access Key](https://app.saucelabs.com/user-settings)
- Access to Sauce Labs Real Devices. Sauce Labs only supports XCTests on Real Devices, not virtual.
- Flutter mobile app. If you don't have one, you could use our Flutter Demo App:
- [Sauce Labs Flutter Demo App](https://github.com/saucelabs/my-demo-app-flutter)
- `xcodebuild` tools
- `zip` and/or `saucectl`
:::


## 1. How to build the '.app' and '.xctestrun' files for your Flutter app.

Check warning on line 38 in docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md#L38

[sauce.HeadingPunctuation] Don't use periods or exclamation marks in headings.
Raw output
{"message": "[sauce.HeadingPunctuation] Don't use periods or exclamation marks in headings.", "location": {"path": "docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md", "range": {"start": {"line": 38, "column": 73}}}, "severity": "WARNING"}

:::note You need to setup your Flutter app for integration tests.

Before you build your app, you must ensure that you correctly set up the `integration_tests` for your flutter-ios app. You can follow the [flutter documentation](https://github.com/flutter/flutter/tree/main/packages/integration_test#integration_test) to do so. The most relevant section is the part on [iOS Device Testing](https://github.com/flutter/flutter/tree/main/packages/integration_test#ios-device-testing). You can stop following Flutter's guide after you have executed the `xcodebuild build-for-testing` command. This command will generate the `.app` and `.xctestrun` files.

Check warning on line 42 in docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md#L42

[sauce.WordList] Use 'iOS' instead of 'ios'.
Raw output
{"message": "[sauce.WordList] Use 'iOS' instead of 'ios'.", "location": {"path": "docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md", "range": {"start": {"line": 42, "column": 395}}}, "severity": "WARNING"}
:::

To execute your xctest, we require your app (which must be packaged together with your XCTests) in '.app' or '.ipa' format. Additionally we need your your `.xctestrun` file, which is the config for your test.

Check warning on line 45 in docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md#L45

[sauce.Inclusivity] Avoid problematic language. Use 'run' instead of 'execute'.
Raw output
{"message": "[sauce.Inclusivity] Avoid problematic language. Use 'run' instead of 'execute'.", "location": {"path": "docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md", "range": {"start": {"line": 45, "column": 4}}}, "severity": "WARNING"}

Check warning on line 45 in docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md#L45

[sauce.Repetition] Remove this duplicate word: 'your'.
Raw output
{"message": "[sauce.Repetition] Remove this duplicate word: 'your'.", "location": {"path": "docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md", "range": {"start": {"line": 45, "column": 146}}}, "severity": "WARNING"}

By default, Xcode will not persist the `.xctestrun` file if you kick off an XCTest on your development machine. To persist the `.xctestrun` file we need to use the `xcodebuild build-for-testing` command. Make sure you are using the correct `scheme` so it includes your integration tests.

```shell
# Example of the xcodebuild command to build the application.
# You will need to adjust the args according to your app.
output="../build/ios_integ"
xcodebuild build-for-testing \
-workspace Runner.xcworkspace \
-scheme Runner \
-xcconfig Flutter/Release.xcconfig \
-configuration Release \
-derivedDataPath \
$output -sdk iphoneos

# The .app and .xctestrun files will now be present in your output directory. In this case: `build/ios_integ/Products/Release-iphoneos`
```


## 2. How to run flutter-iOS integration tests on Sauce Labs infrastructure

Check warning on line 65 in docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md#L65

[sauce.Headings] '2. How to run flutter-iOS integration tests on Sauce Labs infrastructure' should use title case capitalization.
Raw output
{"message": "[sauce.Headings] '2. How to run flutter-iOS integration tests on Sauce Labs infrastructure' should use title case capitalization.", "location": {"path": "docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md", "range": {"start": {"line": 65, "column": 4}}}, "severity": "WARNING"}

To run your Flutter XCTest on Sauce Labs, you have two options: use `saucectl` or integrate with our APIs yourself. If you are unfamiliar with our APIs, we recommend using `saucectl` for ease of use and getting you started quickly.

### Run XCTests via saucectl

Check warning on line 69 in docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md#L69

[sauce.Headings] 'Run XCTests via saucectl' should use title case capitalization.
Raw output
{"message": "[sauce.Headings] 'Run XCTests via saucectl' should use title case capitalization.", "location": {"path": "docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md", "range": {"start": {"line": 69, "column": 5}}}, "severity": "WARNING"}

First install [saucectl](/docs/dev/cli/saucectl.md#installing-saucectl). Then you can use `saucectl` command to configure and run your test on Sauce Labs infrastructure.

```shell
# If it's the first time you're using saucectl, run:
saucectl configure

# follow the steps to configure your XCTest, with your `.app`/`.ipa` file and the `.xctestrun` test config. Use `Real Device` not `Virtual Device`
saucectl init xctest
si-net marked this conversation as resolved.
Show resolved Hide resolved

# run the newly created XCTest config.
saucectl run
```

For further configuration options and info on how to use `saucectl` visit [/docs/mobile-apps/automated-testing/espresso-xcuitest/xcuitest.md](/docs/mobile-apps/automated-testing/espresso-xcuitest/xcuitest.md)

Check warning on line 84 in docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md#L84

[sauce.WordList] Use 'Espresso' instead of 'espresso'.
Raw output
{"message": "[sauce.WordList] Use 'Espresso' instead of 'espresso'.", "location": {"path": "docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md", "range": {"start": {"line": 84, "column": 112}}}, "severity": "WARNING"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should change the saucectl docs to reflect xctests too, this is something that can make customer confused and open SRs.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we need to find a smart way to do this, I don't want to copy the entire xcuitest config page

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about this more, I believe we should not update the doc in the xcuitest config page until we at least support format version 2 of the xctestrun file.

Otherwise people will assume that we have generic XCTest support and wonder when their XCTests don't work on Sauce.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah the confusion would come from the replacement from appTest to xcTestRunFile parameter in the file and the saucectl init command.

Maybe in the part we recommend using saucectl init we could give a heads up there. Like You'll need to provide the test app or ipa file, and the xctestrun file created as described previously. saucectl init can be invoked interactive or non interactive. If using non interactive these are the minimum required parameters:

saucectl init --no-prompt -r us-west-1 xctest --xctest-run-file FlutterTest.xctestrun --app FlutterTestApp-resigned.ipa --device name=iPhone,private=true

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mention that in this comment: follow the steps to configure your xctest, with your .app/.ipafile and the.xctestruntest config. UseReal DevicenotVirtual Device``


### Run XCTests without saucectl

Check warning on line 86 in docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md#L86

[sauce.Headings] 'Run XCTests without saucectl' should use title case capitalization.
Raw output
{"message": "[sauce.Headings] 'Run XCTests without saucectl' should use title case capitalization.", "location": {"path": "docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md", "range": {"start": {"line": 86, "column": 5}}}, "severity": "WARNING"}

If you prefer not to use saucectl, you can directly integrate with our APIs.

**First**, compile your `.app` as an `.ipa` file as described [above](/docs/mobile-apps/automated-testing/ipa-files.md#building-an-ipa-from-an-app-bundle).

Check warning on line 90 in docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md#L90

[sauce.EOLWhitespace] Remove whitespace characters from the end of the line.
Raw output
{"message": "[sauce.EOLWhitespace] Remove whitespace characters from the end of the line.", "location": {"path": "docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md", "range": {"start": {"line": 90, "column": 156}}}, "severity": "WARNING"}

**Second**, upload your `.ipa` and `.xctestrun` files to our AppStorage backend, see [AppStorage APIs](/docs/mobile-apps/app-storage.md#upload-apps-via-rest-api).

**Third**, call our native testing API with the AppStorage IDs of the two files you just uploaded. See [RDC native /test API](/docs/dev/api/rdc.md#start-a-xctest-xcuitest-or-espresso-job).

**Fourth**, poll the state of the job and wait until the `status` is `passed|failed|error|complete`. You can do this through the [Jobs API](/docs/dev/api/rdc.md#get-a-specific-real-device-job).


## Example Implementation

For a practical example of how to set up and run integration tests for Flutter apps, you can refer to
the [Sauce Labs Flutter demo application](https://github.com/saucelabs/my-demo-app-flutter) repository.
The steps outlined in this guide have already been implemented in that repository. You can follow along with the demo app to see how
everything is configured and run your tests accordingly.
7 changes: 7 additions & 0 deletions docs/mobile-apps/automated-testing/ipa-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ In order to disable the resigning process, you must buy your own private devices
3. Compress the `Payload` directory into an archive (.zip file) and give it a new name with .ipa appended to the end of the file name.
4. Your `.ipa` file is now ready for upload to Sauce Labs.

```shell
# example for building an '.ipa' file out of an '.app'
mkdir Payload
cp -r PATH_TO_BUILD_FOLDER/Runner.app Payload
zip -r Runner.ipa Payload
```

### Building an .ipa File

You can use any of the existing methods of distribution for your iOS app, except for the **App Store** type. This means that you can choose any of the three other export methods: **Ad Hoc**, **Enterprise**, or **Development**.
Expand Down
3 changes: 2 additions & 1 deletion sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -1438,6 +1438,7 @@ module.exports = {
items: [
'mobile-apps/automated-testing/flutter',
'mobile-apps/automated-testing/flutter/flutter-integration-testing-android',
'mobile-apps/automated-testing/flutter/flutter-integration-testing-ios',
],
},
'mobile-apps/automated-testing/ipa-files',
Expand Down Expand Up @@ -1728,7 +1729,7 @@ module.exports = {
'visual-testing/integrations/python',
'visual-testing/integrations/python-robot-framework',
'visual-testing/integrations/playwright',
'visual-testing/integrations/espresso'
'visual-testing/integrations/espresso',
],
},
'visual-testing/cli',
Expand Down
Loading