[WIP] Add Support for PodBuilder

[fabb]

As discussed in tmspzz#155, support for PodBuilder would be nice additionally to Carthage.

⚠️ Not finished, this is work in progress.

• Parse and analyze the Podfile.restore file to find out which framework versions the project depends on
• Parse and analyze the PodBuilder.podspec file to find out which frameworks would need to be generated
• Parse the PodBuilder.plist files and use them instead of Carthage version files
• Parse the pod_builder info json output from a file
• Receive podbuilder info output via STDIN or as a parameter (a temporary file would not be so nice)
• Add and parse a configuration option flag which tells Rome that it needs to deal with a PodBuilder project
• (optional) put the configuration option to Romefile
• Add and parse a configuration option flag for passing the compiler version, since pod_builder info contains the swift_version for each built Swift framework, and we need to compare it to the expected one, and use it as part of the cache key
• Extract functionality common to Carthage and PodBuilder, and execute different code paths based on configuration option
• Use the PodBuilder folder structure instead of the one of Carthage/Build
• Since CocoaPods/Rome and therefore PodBuilder only seems to support one platform at a time, ignore TargetPlatform, or set it to some placeholder
• Update Readme

[fabb]

@blender I implemented the parser for Podfile.restore 🎉. I would be grateful if you could look over PodfileRestore.hs, and give me directions for these questions:

• In the parser, I just parsed lines starting with pod (and that end with # pb<...>), I think this should be ok since PodBuilder builds all frameworks from all targets anyways into a flat list. Sounds reasonable?
• In the Podfile.restore, subspec references like AFNetworking/NSURLSession can occur, but PodBuilder will only build a single framework. I have the hunch that I should use the set of subspecs as a part of the framework version, so it cannot happen that the client project adds a subspec for a pod in its Podfile, but does not change the version of the pod, and Rome serves the cached framework, but missing the parts of the added subspec. I need to check with the maintainer of PodBuilder. What do you think?
• I have not filtered out duplicates across targets or subspecs, I guess this can be done at a later point when the data from the parsed Podfile.restore and PodBuilder.podspec are combined?
• PodBuilder only allows to build for one platform at a time. It might still be interesting to store the frameworks in a folder with the platform in the cache. I could parse the line starting with platform from Podfile.restore, but I was not sure how to parse it properly. Any hints?
• Do the data structures PodfileEntry, PodName and PodVersion make sense like this for further processing?
• Should I write some unit tests? Due to Parsec.try parseMaybePodfileEntry, parsing errors are silently swallowed, which makes the parser a little fragile. Should I add a stripped down Podfile.restore to Tests/Fixtures/, and write an HSpec test?

Unfortunately I found out that I also need to parse PodBuilder.podspec in order to find out how the framework folders are named... So later I need to combine the information with the parsed one from Podfile.restore. But one after the other. At least I have a next step ;-)

[tmspzz]

I just parsed lines starting with pod (and that end with # pb<...>)

Sounds reasonable

In the Podfile.restore, subspec references like AFNetworking/NSURLSession can occur, but PodBuilder will only build a single framework.

I think you should treat this independently of what Rome does. So if you need to track subspecs for the sake of correctly parsing/processing the Podbuilder information do so. Then for Rome you can construct something else that collapses (or filters out) the duplicate information accordingly.

I have not filtered out duplicates across targets or subspecs, I guess this can be done at a later point when the data from the parsed Podfile.restore and PodBuilder.podspec are combined?

The parser should just do the parsing. Deal with duplicates later.

I could parse the line starting with platform from Podfile.restore, but I was not sure how to parse it properly. Any hints?

I'm eyeballing it so don't assume this works

Parsec.spaces 
>> Parsec.string "platform"  -- parses "platform"
>> Parsec.spaces 
*> Parsec.between (Parsec.symbol ":") (Parsec.symbol ",")  -- parses " :whateverstring, "
<* Parsec.spaces
>>  Parsec.between (Parsec.symbol "'") (Parsec.symbol "'") -- parses " 'whateverstring' "

Do the data structures PodfileEntry, PodName and PodVersion make sense like this for further processing?

They look good to me, if you want to parse the platform add PodPlatform

Should I write some unit tests? Due to Parsec.try parseMaybePodfileEntry, parsing errors are silently swallowed, which makes the parser a little fragile. Should I add a stripped down Podfile.restore to Tests/Fixtures/, and write an HSpec test?

Yes, add as many as fixtures as you want, they don't have to be stripped down. Having the real deal is better. I don't recall how I do the error reporting for the Cartfile but I guess that it should be the same.

[tmspzz]

this module name seems off

[fabb]

@blender Thanks! But change of plans: I talked to @tcamin (maintainer of PodBuilder, actually he works in a sister company), and we found out that even when parsing Podfile.restore and PodBuilder.podspec there are still cases where we cannot infer the framework file names without additional information, e.g. when using different subspecs in app extensions (see subspecs_to_split documentation of PodBuilder). As a solution for integrating blender/Rome, @tcamin introduced a new pod_builder info command which will output a nicely formatted json with all the information that we need:

$ pod_builder info
{
  "AFNetworking": {
    "podbuilder_name": "AFNetworking",
    "framework_path": "Rome/AFNetworking.framework",
    "restore_info": {
      "version": {
        "repo": "https://github.com/willhaben/AFNetworking.git",
        "hash": "c683ebf"
      },
      "specs": [
        "AFNetworking/NSURLSession",
        "AFNetworking/Reachability",
        "AFNetworking/Security",
        "AFNetworking/Serialization",
        "AFNetworking/UIKit"
      ]
    }
  },
  "AGWindowView": {
    "podbuilder_name": "AGWindowView",
    "framework_path": "Rome/AGWindowView.framework",
    "restore_info": {
      "version": {
        "repo": "https://github.com/willhaben/AGWindowView.git",
        "hash": "cef7821"
      },
      "specs": [
        "AGWindowView"
      ]
    },
    "prebuilt_info": {
      "version": {
        "repo": "https://github.com/willhaben/AGWindowView.git",
        "hash": "cef7821"
      },
      "specs": [
        "AGWindowView"
      ]
    }
  },
  "ATInternet-Apple-SDK": {
    "podbuilder_name": "ATInternet-Apple-SDK",
    "framework_path": "Rome/Tracker.framework",
    "restore_info": {
      "version": {
        "tag": "2.9.3"
      },
      "specs": [
        "ATInternet-Apple-SDK/Tracker"
      ]
    },
    "prebuilt_info": {
      "swift_version": "swiftlang-1000.11.42",
      "version": {
        "tag": "2.9.3"
      },
      "specs": [
        "ATInternet-Apple-SDK/Tracker"
      ]
    }
  }
}

So I will parse this output instead of Podfile.restore, PodBuilder.podspec and all the PodBuilder.plist files. (I guess the Aeson Haskell lib will make this easy). The key prebuilt_info contains info about the currently present compiled frameworks (similar to Carthages .version files).

So parsing this should be easy, but getting it into blender/Rome not, since it's not stored in a file. I thought about STDIN, what do you think? I'd like to avoid temporary files.

[tmspzz]

I think your best option is first going via file. Once successful we can think of how to use stdin.

[fabb]

@blender now that was easy, parsing of the json is done. My next steps would be to add a configuration option to the Romefile to make rome know it deals with a PodBuilder project, and depending on this execute different code paths. And extract common functionality to both Carthage and PodBuilder project handling (I'm still very vague about this, need to understand the code better first). If you have any directions, very welcome!

[tmspzz]

Ok

first of a few questions:

• Is there a need for the repositoryMap with PodBuilder?
• Is there a need for the ignoreMap ? (I guess, yes)
• Is there a need for the currentMap ? (This is for the current framework currently being built)

I would start by adding a CLI option with default value carthage instead of a section in the config file like:

rome upload --build-type=carthage or rome upload --build-type=podbuilder

(Maybe there is a more appropriate name for the flag)

you can then analyzer the CLI flag right in the runRomeWithOptionsmethod and branch there.

The function to upload and download and upload to S3 are pretty generic and agnostic of Carthage (or so I recall).

The main trick is a few HashMap that store:

repository name -> binary name
binary name -> repository name (InvertedRepositoryMap)

I would ignore any networking to start with and just do the local filesystem caching for now.

[tmspzz]

I think this should be Data.Map.Strict

[tmspzz]

MonadIO m => m (Maybe PodBuilderInfo)

[tmspzz]

I think there is a withFile that will close the descriptor. I might be wrong.

[fabb]

i think it should be ok:

Because we don't get a handle with which to identify our file, we can't close it manually, so Haskell does that for us when we use readFile.
http://learnyouahaskell.com/input-and-output

[tmspzz]

My bad

[fabb]

• Is there a need for the repositoryMap with PodBuilder?
• Is there a need for the ignoreMap ? (I guess, yes)
• Is there a need for the currentMap ? (This is for the current framework currently being built)

I think I need neither. The repositoryMap and the ignoreMap are provided implicitly by pod_builder info since it already provides the output framework names + it does not include frameworks anyways that do not produce a framework (e.g. R.swift).

I'm not sure about the currentMap since I never thought about using PodBuilder for a framework project, I'd leave this out for now, and come back to it if it gets relevant.

I would start by adding a CLI option with default value carthage instead of a section in the config file like:

rome upload --build-type=carthage or rome upload --build-type=podbuilder

Ok done. I also added a compiler-version CLI option, since pod_builder info already provides valuable info about the compiler version of built frameworks, and I can use it as part of the cache key - it's more sophisticated than using the Swift compiler version as a general cache-prefix.

you can then analyzer the CLI flag right in the runRomeWithOptionsmethod and branch there.

Ok, next up :-)

[tmspzz]

I think you still need the ignoreMap.

All is nice till a framework has a .modulemap for a system library and the binaries are not relocatable.

[tmspzz]

I also added a compiler-version CLI option

mmm, can you show me the code? I'm not sure what the purpose of this would be since you can populate cache-prefix with for example swift_4_2_1-swiftlang-1000_11_42-clang-1000_11_45_1

[tmspzz]

Also you might want to pull from my repo. I changed a few things and now the build is much faster.

[fabb]

I think you still need the ignoreMap.

All is nice till a framework has a .modulemap for a system library and the binaries are not relocatable.

Hm good hint, do you have an example so I can try with PodBuilder?

mmm, can you show me the code? I'm not sure what the purpose of this would be since you can populate cache-prefix with for example swift_4_2_1-swiftlang-1000_11_42-clang-1000_11_45_1

Not every framework is a Swift framework, so not all need the common cache prefix. PodBuilder provides the swift_version only for Swift frameworks, see

Rome/PodBuilderInfo.json

Line 60 in 6f1ca1d

"swift_version": "swiftlang-1000.11.42",

[tmspzz]

Non-relocatable framework example: https://github.com/daltoniam/Starscream

not all need the common cache prefix. PodBuilder provides the swift_version only for Swift frameworks

Sure. So you plan to upload/download each framework from the appropriate location in the Cache based on this information?

I see how this can be valuable to reduce duplication in the cache but I feel like it's a bit of an overkill and also makes the current cache structure incompatible between Carthage/Podbuilder.

It would be better in my opinion not to alter the cache structures. Why? Because in the end it should not matter if the uploader was using Carthage or Podbuilder

[fabb]

Sure. So you plan to upload/download each framework from the appropriate location in the Cache based on this information?

Exactly, I‘d put it into the filename additionally to the version-hash.

Just hypothetically if I used the cache prefix instead, how would I in Rome check if the local frameworks need to be replaced by ones of the cache (because we don‘t know which Swift version The local frameworks have, and which would be expected)?

It would be better in my opinion not to alter the cache structures. Why? Because in the end it should not matter if the uploader was using Carthage or Podbuilder

While a good idea in theory, this won‘t work in practice anyways, even when I would ignore the swift_version. Cocoapods supports subspecs, that means a built framework would only include one of several different sets of subspecs. The cached framework should only be used for exactly the same set of expected subspecs. Therefore I‘d also put the subspec names (or a hash of the array of subspec names) into the filename.

[tmspzz]

Just hypothetically if I used the cache prefix instead, how would I in Rome check if the local frameworks need to be replaced by ones of the cache

Rome doesn't know things need to be replaced. It's the responsibility of the user to enter the right prefix (maybe by invoking rome like rome download --cache-prefix=`xcrun swift --version` and download dependencies when they change.

Rome is just a vector (a transport system) to the cache. Rome knows where to get and where to put binaries at, but that's all. It's pretty dumb. I could make it smarter but I don't know if I want to take the freedom away from the user and put the burden on the tool on when and where do things. The more I automate the more error prone the process is.

More specifically:

Carthage produces a Cartfile.resolved. In this file the commit hash or tag of the dependency is saved once it has been resolved at the right version. When the user does rome upload, the Carfile.resolved is read, the frameworks in Carthage/Build/<platform> are uploaded along side with their .bcsymbolmaps and .version files.

They are upload at a path constructed in the following way:

Frameworks and dSYMs: <git-repository-name>/<platform>/<framework-name>.framework(.dSYM)-(static-)<version-hash>.zip
bcsymbolmaps: <git-repository-name>/<platform>/<bcsymbolmap-hash>.bcsymbolmap-(static-)<version-hash>.zip
.version files: <git-repository-name>/.<framework-name>.version-(static-)<version-hash>

When the user runs rome list, the Cartfile.resolved is read, the path are constructed in the same way and the cache is queried for a hit or a miss.
When the user runs rome download, the Cartfile.resolved is read, the path are constructed in the same way and binaries downloaded, no questions asked.

Alone Rome won't do anything for you. However you can combine command to make it smarted (Functional programming strikes again in the design :D), for example:

rome list --missing --platform iOS | awk '{print $1}' | xargs rome upload --platform iOS

to upload only what is not already in the caches.

In theory I could read the .version files to know if a framework is objective-c, what version of the compiler it was built with and so on... but:

1. The .version files are part of the cache, so Rome would have to maintains it's own database because on clean checkout of a project the .version files are not there (they are in the cache!)
2. Carthage doesn't produce .version files by default. It's an CLI option (--cache-builds)

While a good idea in theory, this won‘t work in practice anyways

I'm not sure I understand why. Maybe you can help me? A binary is a binary and to Carthage (at least) all that it matters is that it's placed in the right directory. I imagine Podbuilder also expects binaries to be at a certain location on disk so it's just a matter of placing them there on the user's machine. I'm not sure why where they are in the cache would matter.

Exactly, I‘d put it into the filename additionally to the version-hash

That would also break the compatibility of the caches.

[fabb]

Hm, maybe I missed a piece: until now I thought rome download would only download frameworks that were not up to date, but now I rather think it always downloads everything. Does it?

In theory I could read the .version files to know if a framework is objective-c, what version of the compiler it was built with and so on... but:

1. The .version files are part of the cache, so Rome would have to maintains it's own database because on clean checkout of a project the .version files are not there (they are in the cache!)

Good point, also in the case of PodBuilder I currently do not have the information a priori whether it will be a Swift framework or not. Hm.

I'm not sure I understand why. Maybe you can help me? A binary is a binary and to Carthage (at least) all that it matters is that it's placed in the right directory. I imagine Podbuilder also expects binaries to be at a certain location on disk so it's just a matter of placing them there on the user's machine. I'm not sure why where they are in the cache would matter.

The same framework name could be produced by completely different source code, because of subspecs. For example see https://github.com/AFNetworking/AFNetworking/blob/master/AFNetworking.podspec. When I build AFNetworking just with subspec NSURLSession for one project, I cannot reuse that framework for another project that needs the UIKit subspec.

Exactly, I‘d put it into the filename additionally to the version-hash

That would also break the compatibility of the caches.

I would not change how the filenames were constructed for in Carthage mode of course.

[tmspzz]

but now I rather think it always downloads everything. Does it?

Yes. In theory I could limit what it downloads by looking at the .versions files if any. That's actually a good point.

On the first run the user doesn't have any, so download everything. On subsequent runs check what versions files are out of sync with the truth (the Cartfile.resolved).

I never thought of this. Thanks for the inspiration! 😄

The same framework name could be produced by completely different source code, because of subspecs.

so a subspec is a portion of a complete framework built on demand? If this is the case then I understand why the caches binaries in the cache can't be shared.

[fabb]

I never thought of this. Thanks for the inspiration! 😄

Molto volentieri! BUT: this won‘t work in the case when using the cache prefix for the Swift version, because you have no way to verify if the local framework stems from the cache with the correct prefix. That‘s why I thought I need to give more information to the dumb vector.

so a subspec is a portion of a complete framework built on demand? If this is the case then I understand why the caches binaries in the cache can't be shared.

Yes, with the exception that there might even be mutually exclusive subspecs. E.g. https://github.com/at-internet/atinternet-apple-sdk/blob/master/ATInternet-Apple-SDK.podspec contains separate subspecs for the main target and the app extension. In this case, PodBuilder even builds two differently postfixed frameworks. See https://github.com/Subito-it/PodBuilder/blob/master/README.md#subspecs_to_split for more information.

[tmspzz]

this won‘t work in the case when using the cache prefix for the Swift version, because you have no way to verify if the local framework stems from the cache with the correct prefix.

I'm not sure why I would need to verify the Swift version. Rome doesn't know about Swift versions to begin with. The Swift version handling is artificial and manual based on the prefix which is user input.

Nothing presents having a cache prefix like swift_4.2, and someone uploading a framework built for 4.0 there. The prefix is just a path appended to the root of the cache. For example ~/Library/Caches/Rome/swift_4.2 or ~/Library/Cache/Rome/mike-test-unstripped where ~/Library/Caches/Rome is the base path to the cache.

What it can do however is check what it needs to download based on cross referencing the hashes in the Cartfile.resolved and in the .version file for each framework in Carthage/Build/<platform>/iOS

[fabb]

you can then analyzer the CLI flag right in the runRomeWithOptions method and branch there.

Instead of branching, I tried to introduce a sum type BuildTypeSpecificConfiguration that allows to share more logic between Carthage and PodBuilder functionality, see the latest commit. Note that currently even though it compiles, and :main download --build-type podbuilder runs, still the file paths are not built correctly, and .version file still should be ignored. Getting there. also there are a lot of TODOs in deriveFrameworkNamesAndVersionPodBuilder.
What do you think of this approach?

this won‘t work in the case when using the cache prefix for the Swift version, because you have no way to verify if the local framework stems from the cache with the correct prefix.

I'm not sure why I would need to verify the Swift version. Rome doesn't know about Swift versions to begin with. The Swift version handling is artificial and manual based on the prefix which is user input.

Nothing presents having a cache prefix like swift_4.2, and someone uploading a framework built for 4.0 there. The prefix is just a path appended to the root of the cache. For example ~/Library/Caches/Rome/swift_4.2 or ~/Library/Cache/Rome/mike-test-unstripped where ~/Library/Caches/Rome is the base path to the cache.

What it can do however is check what it needs to download based on cross referencing the hashes in the Cartfile.resolved and in the .version file for each framework in Carthage/Build/<platform>/iOS

Maybe I'm somehow wrong, but consider this example:
Given the download command is already modified to only download out of sync frameworks, using version files and the Cartfile.resolved.

Scenario 1:

• Carthage/Pods contains FrameworkX.framework built with Swift 4
• The .version file of FrameworkX matches with what is specified in Cartfile.resolved
• The cache contains both swift4/FrameworkX/iOS/FrameworkX.framework-1.0.0.zip and swift3/FrameworkX/iOS/FrameworkX.framework-1.0.0.zip (built with the respectively correct Swift versions)
• The currently expected Swift version is 4
• The currently set cache key is swift4
  => ✅ download would not fetch FrameworkX.framework, as it's up to date

Scenario 2:

• Carthage/Pods contains FrameworkX.framework built with Swift 4
• The .version file of FrameworkX matches with what is specified in Cartfile.resolved
• The cache contains both swift4/FrameworkX/iOS/FrameworkX.framework-1.0.0.zip and swift3/FrameworkX/iOS/FrameworkX.framework-1.0.0.zip (built with the respectively correct Swift versions)
• The currently expected Swift version is 3
• The currently set cache key is swift3
  => ❌ download would not fetch FrameworkX.framework, even though the currently present one was built with the wrong Swift version. The reason is that Rome does not know which Swift version and cache prefix the present framework is associated with, as there is no info about this in the .version or Cartfile.resolved files. It does not know enough about the present frameworks to judge if they are out of sync, the .version file is not detailed enough.

I currently can't really think of a way to give Rome this information in a more abstract way than to teach it about the Swift version. Sure, I could give up on the optimization of not downloading everything, but where would the fun be in that? 😄

[tmspzz]

Oh I see.

Sure scenario 2 won't work, however I don't know how common this is. It seems like an overkill to me to worry about the same commit building for 2 different versions of Swift and people deciding to pull in one from an older version of the compiler. If the user wants to do so I think it should be his responsibility non to run the optimized download/upload.

The current download/upload is not optimized anyways so that would have to be another CLI option like:

rome (download|upload) --platform iOS Alamofire --if-needed --cache-prefix=my-prefix

[tmspzz]

What do you think of this approach?

Seems good. Great work! 🎉

[fabb]

Ok, slowly getting there. The generated paths are not yet correct, but now I have the BuildTypeSpecificConfiguration everywhere so I can create different paths for Carthage and PodBuilder. I'm a bit ashamed of putting the additional BuildTypeSpecificConfiguration parameter in so many methods 🙈 If you have any suggestions, keep them coming ;-)

[fabb]

rebased everything on the latest master, phew, this was quite some work! 😅
i'm already a bit afraid of tmspzz#148 😆

[fabb]

Ok, moved all the temp_ functions to FrameworkVector, next I can remove all the now unnecessary buildTypeConfig parameters in the rest of the application again, and use different paths for Carthage and PodBuilder.

[fabb]

Since SwiftPM is now integrated into Xcode 11, I will stop developing this PR. Also, the implementation strategy conflicts massively with tmspzz#148, and a lot of work would need to be put in to resolve this issue, and additionally the issue of supporting both device and simulator builds.
It was a nice fresh up of my Haskell skills though. Thanks @blender for your guidance.

[tmspzz]

oh damn! Too bad :( I hope you had a nice time playing around with Haskell though!