Hi @mthmulders, I am currently analysing the issue and have successfully reproduced it. I will keep you updated on my progress.

This seems like a Class loader problem in Liberty . Tagging @tjwatson and @dazavala for further insight.

The issue is the jpa-1.jar with the class it.mulders.ol.jpa.ThingEntity is still contained in each WAR's WEB-INF/lib folder making it not shared between the two WARs. The issue is the jpa dependency is not declared for the EAR's pom.xml to get it properly placed int he EAR's lib/ folder.

See my PR to your repo: mthmulders/openliberty-multi-app-deploy#1

Thank you @ajaypaul-ibm and @tjwatson for your investigations.

@tjwatson, I've merged your suggestion, ran mvn clean, then mvn package and finally mvn liberty:dev.

After mvn package, I verified that the jpa-1.jar is now in the EAR's lib/ folder, and that both JAR's don't have it included anymore:

• Content of unzipped EAR

  $ tree
  .
  ├── META-INF
  │   ├── MANIFEST.MF
  │   ├── application.xml
  │   └── maven
  │       └── it.mulders.ol
  │           └── deployment
  │               ├── pom.properties
  │               └── pom.xml
  ├── it.mulders.ol-api-1.war
  ├── it.mulders.ol-webapp-1.war
  └── lib
      ├── ch.qos.logback-logback-classic-1.5.12.jar
      ├── ch.qos.logback-logback-core-1.5.12.jar
      ├── it.mulders.ol-domain-1.jar
      ├── it.mulders.ol-jpa-1.jar
      └── org.slf4j-slf4j-api-2.0.16.jar
  
  6 directories, 11 files
  

• Inspecting the WAR's inside the EAR

  $ find . -name "*.war" -exec unzip -l {} \;
  
  Archive:  ./it.mulders.ol-api-1.war
    Length      Date    Time    Name
  ---------  ---------- -----   ----
        283  02-01-1980 00:00   META-INF/MANIFEST.MF
          0  02-01-1980 00:00   META-INF/
          0  02-01-1980 00:00   WEB-INF/
          0  02-01-1980 00:00   WEB-INF/classes/
          0  02-01-1980 00:00   WEB-INF/classes/it/
          0  02-01-1980 00:00   WEB-INF/classes/it/mulders/
          0  02-01-1980 00:00   WEB-INF/classes/it/mulders/ol/
          0  02-01-1980 00:00   WEB-INF/classes/it/mulders/ol/api/
          0  02-01-1980 00:00   WEB-INF/lib/
        288  02-01-1980 00:00   WEB-INF/beans.xml
        420  02-01-1980 00:00   WEB-INF/classes/it/mulders/ol/api/ExampleApplication.class
        712  02-01-1980 00:00   WEB-INF/classes/it/mulders/ol/api/RandomNumberGenerator.class
       1696  02-01-1980 00:00   WEB-INF/classes/it/mulders/ol/api/StatusDTO.class
       1978  02-01-1980 00:00   WEB-INF/classes/it/mulders/ol/api/StatusResource.class
        366  02-01-1980 00:00   WEB-INF/classes/logback.xml
       1735  02-01-1980 00:00   META-INF/maven/it.mulders.ol/api/pom.xml
         47  02-01-1980 00:00   META-INF/maven/it.mulders.ol/api/pom.properties
  ---------                     -------
       7525                     17 files
  Archive:  ./it.mulders.ol-webapp-1.war
    Length      Date    Time    Name
  ---------  ---------- -----   ----
        283  02-01-1980 00:00   META-INF/MANIFEST.MF
          0  02-01-1980 00:00   META-INF/
          0  02-01-1980 00:00   WEB-INF/
          0  02-01-1980 00:00   WEB-INF/classes/
          0  02-01-1980 00:00   WEB-INF/classes/it/
          0  02-01-1980 00:00   WEB-INF/classes/it/mulders/
          0  02-01-1980 00:00   WEB-INF/classes/it/mulders/ol/
          0  02-01-1980 00:00   WEB-INF/classes/it/mulders/ol/webapp/
          0  02-01-1980 00:00   WEB-INF/lib/
        288  02-01-1980 00:00   WEB-INF/beans.xml
       1457  02-01-1980 00:00   WEB-INF/classes/it/mulders/ol/webapp/IndexView.class
        366  02-01-1980 00:00   WEB-INF/classes/logback.xml
        353  02-01-1980 00:00   WEB-INF/faces-config.xml
        407  02-01-1980 00:00   index.xhtml
       1393  02-01-1980 00:00   web.xml
       2065  02-01-1980 00:00   META-INF/maven/it.mulders.ol/webapp/pom.xml
         50  02-01-1980 00:00   META-INF/maven/it.mulders.ol/webapp/pom.properties
  ---------                     -------
       6662                     17 files
  

But alas, I am still facing the same error message.

Turns out this is now a liberty plugin issue with the generated loose application XML. The liberty maven plugin is not aware of the <skinnyModules>true</skinnyModules> configuration for the production of the EAR. This means the generation of the file target/liberty/wlp/usr/servers/defaultServer/apps/multi-app.ear.xml is not correct. For example, see the section for the it.mulders.ol-api-1.war WAR

    <archive targetInArchive="/it.mulders.ol-api-1.war">
        <dir sourceOnDisk="/home/tjwatson/dev/liberty/examples/openliberty-multi-app-deploy/api/src/main/webapp" targetInArchive="/"/>
        <dir sourceOnDisk="/home/tjwatson/dev/liberty/examples/openliberty-multi-app-deploy/api/target/classes" targetInArchive="/WEB-INF/classes"/>
        <file sourceOnDisk="/home/tjwatson/dev/liberty/examples/openliberty-multi-app-deploy/api/target/tmp/META-INF/MANIFEST.MF" targetInArchive="/META-INF/MANIFEST.MF"/>
        <archive targetInArchive="/WEB-INF/lib/domain-1.jar">
            <dir sourceOnDisk="/home/tjwatson/dev/liberty/examples/openliberty-multi-app-deploy/domain/target/classes" targetInArchive="/"/>
            <file sourceOnDisk="/home/tjwatson/dev/liberty/examples/openliberty-multi-app-deploy/domain/target/tmp/META-INF/MANIFEST.MF" targetInArchive="/META-INF/MANIFEST.MF"/>
        </archive>
        <archive targetInArchive="/WEB-INF/lib/jpa-1.jar">
            <dir sourceOnDisk="/home/tjwatson/dev/liberty/examples/openliberty-multi-app-deploy/jpa/target/classes" targetInArchive="/"/>
            <file sourceOnDisk="/home/tjwatson/dev/liberty/examples/openliberty-multi-app-deploy/jpa/target/tmp/META-INF/MANIFEST.MF" targetInArchive="/META-INF/MANIFEST.MF"/>
        </archive>
        <file sourceOnDisk="/home/tjwatson/.m2/repository/org/slf4j/slf4j-api/2.0.16/slf4j-api-2.0.16.jar" targetInArchive="/WEB-INF/lib/slf4j-api-2.0.16.jar"/>
        <file sourceOnDisk="/home/tjwatson/.m2/repository/ch/qos/logback/logback-classic/1.5.12/logback-classic-1.5.12.jar" targetInArchive="/WEB-INF/lib/logback-classic-1.5.12.jar"/>
        <file sourceOnDisk="/home/tjwatson/.m2/repository/ch/qos/logback/logback-core/1.5.12/logback-core-1.5.12.jar" targetInArchive="/WEB-INF/lib/logback-core-1.5.12.jar"/>
    </archive>

It still has entries for the domain and jpa JARs even though they are also in the lib/ of the EAR.

We will need to move this issue over to liberty maven plugin.

If you want to see it "work" then you can copy the produced EAR file deployment/target/multi-app.ear to deployment/target/liberty/wlp/usr/servers/defaultServer/apps/ and delete the file deployment/target/liberty/wlp/usr/servers/defaultServer/apps/multi-app.ear.xml

Then you can start the server (without using mvn) with:
./deployment/target/liberty/wlp/bin/server run

Thanks for the investigation @tjwatson . Just searching the code quickly I can see that our loose app deployment seems to have support for the <skinnyWars> configuration parm of maven-ear-plugin but NOT for <skinnyModules> .

It seems worth moving the issue over to our liberty-maven-plugin repo (which I did) then to investigate further. Thanks.

@tjwatson @scottkurz thanks for your investigations! I can confirm it works as expected if I don't use the liberty-maven-plugin and manually start OpenLiberty, as suggested by @scottkurz.

@tjwatson @scottkurz thanks for your investigations! I can confirm it works as expected if I don't use the liberty-maven-plugin and manually start OpenLiberty, as suggested by @scottkurz.

And note it is still possible to use liberty-maven-plugin to deploy with loose app disabled, (either -DlooseApplication=false or plugin config <looseApplication>false</looseApplication> ). It's not as convenient, you have to add the goals to the lifecycle yourself instead of using dev mode (or use dev mode with some non-obvious workarounds). If this sounds like something you want to try, though, and you need help with the workaround feel free to let us know.

And note it is still possible to use liberty-maven-plugin to deploy with loose app disabled, (either -DlooseApplication=false or plugin config <looseApplication>false</looseApplication> ). It's not as convenient, you have to add the goals to the lifecycle yourself instead of using dev mode (or use dev mode with some non-obvious workarounds). If this sounds like something you want to try, though, and you need help with the workaround feel free to let us know.

For the time being, I'd love to have some hints on this. I tried writing some convenience scripts to speed up the build + restart Open Liberty cycle, but the feedback loop is considerably shorter than when I use mvn liberty:dev.

From what I understand, I guess it starts with setting looseApplication to false and then running mvn package -pl :artifact-with-ear in a second terminal?

Could you consider using <skinnyWars> for now? That would not make your RARs or EJBs contained directly in the EAR "skinny". In practice I am unsure that is a real problem since EJB and RAR modules in EARs get placed on the EAR class loader already (including their Class-Path references).

Could you consider using <skinnyWars> for now?

I don't use RARs or EJBs in my project, and I don't plan to use them, so it shouldn't be a problem.

However, I've just tried to replace skinnyModules with skinnyWars in the Maven EAR Plugin configuration. Running mvn liberty:dev -pl :deployment -am prints out

[INFO] [WARNING ] CWNEN0047W: Resource annotations on the fields of the it.mulders.ol.webapp.IndexView class will be ignored. The annotations could not be obtained because of the exception : java.lang.NoClassDefFoundError: org/slf4j/Logger
[INFO] [WARNING ] CWNEN0047W: Resource annotations on the fields of the it.mulders.ol.api.StatusResource class will be ignored. The annotations could not be obtained because of the exception : java.lang.NoClassDefFoundError: org/slf4j/Logger

And both applications don't get deployed. The contents of the EAR look good, though, and as you can see, SLF4J is indeed in the lib/ folder of the EAR:

.
├── META-INF
│   ├── MANIFEST.MF
│   ├── application.xml
│   └── maven
│       └── it.mulders.ol
│           └── deployment
│               ├── pom.properties
│               └── pom.xml
├── it.mulders.ol-api-1.war
├── it.mulders.ol-webapp-1.war
└── lib
    ├── ch.qos.logback-logback-classic-1.5.12.jar
    ├── ch.qos.logback-logback-core-1.5.12.jar
    ├── it.mulders.ol-domain-1.jar
    ├── it.mulders.ol-jpa-1.jar
    └── org.slf4j-slf4j-api-2.0.16.jar

6 directories, 11 files

Does that happen with the produced EAR archive also? If so then my understanding of how skinnyWars works for the maven EAR production is not correct and you would need to continue to use skinnyModules. If the EAR archive works vs. the loose application XML then this indicates a separate issue with how the liberty maven plugin handles skinnyWars.

Oops, I noticed I didn't set looseApplication to false an hour ago. Now I did. Still didn't work. After that, I double-checked the generated multi-app.ear.xml and I noticed there were a few entries like this

<archive targetInArchive="/lib//it.mulders.ol-domain-1.jar">

Notice the double / in the path!

I then checked the defaultLibBundleDir parameter for the Maven EAR Plugin and it had the value lib/. I've removed that trailing / and now it seems to be working correctly.

This improves the feedback cycle already, as I now only have to run mvn package -pl :deployment -am in a second terminal and no longer need to restart Open Liberty completely. Still, it's not on par with using mvn liberty:dev alone and having an immediate reload when you change the code.

So to summarise, I went from skinnyModules=true to skinnyWars=true in the Maven EAR Plugin and added looseApplication=false in the Liberty Maven Plugin.

Am I missing something?

So to summarise, I went from skinnyModules=true to skinnyWars=true in the Maven EAR Plugin and added looseApplication=false in the Liberty Maven Plugin.

If you've switched from skinnyModules=true to skinnyWars=true then you should be able to use loose apps (which defaults to true).

This seemed to work fine when I tested your recreate app repo. Though I'm a bit confused because I noticed it seemed to work just fine for me on the initial commit too (back when the jpa-1 JAR was packaged within each WAR). I wonder if there is some other variable or condition in the use case required to recreate the problem, and/or maybe there's something related to the CDI scope in play? It's also possible that my inability to recreate really doesn't matter, ultimately, with any of the points discussed here.. (but thought it was worth mentioning).

Sorry for not getting back earlier.

If you've switched from skinnyModules=true to skinnyWars=true then you should be able to use loose apps (which defaults to true).

I've just tried to do this. In the reproducer that I shared, it seems to work. But in a larger project (which is also open source, but because it's so large I first created a small reproducer) it still doesn't work. As far as I can tell, all relevant Maven and OpenLiberty configuration is the same in both projects.

This seemed to work fine when I tested your recreate app repo. Though I'm a bit confused because I noticed it seemed to work just fine for me on the initial commit too (back when the jpa-1 JAR was packaged within each WAR). I wonder if there is some other variable or condition in the use case required to recreate the problem, and/or maybe there's something related to the CDI scope in play? It's also possible that my inability to recreate really doesn't matter, ultimately, with any of the points discussed here.. (but thought it was worth mentioning).

I've just tried to do this. In the reproducer that I shared, it seems to work. But in a larger project (which is also open source, but because it's so large I first created a small reproducer) it still doesn't work. As far as I can tell, all relevant Maven and OpenLiberty configuration is the same in both projects.

Thanks for responding. OK, fair enough. It just concerned me that we might be missing something without recreating that first step, but in any case, it's clear the loose app deployment for skinnyModules looks incorrect as noted. So since @tjwatson seemed to make the link from this incorrect structure to a potential classloader problem, I think we have enough detail to address this problem in the liberty-maven-plugin deploy code.