[AVM CI Environment Issue]: Deployment step is failing when template size exceeds the maximum size of 4 MB

[kpantos]

Check for previous/existing GitHub issues

• I have checked for previous/existing GitHub issues

Issue Type?

Bug

Description

The deployment part of the CI workflow fails at the Validate Template File step when template size exceeds the maximum size of 4 MB.

WARNING: The request content size exceeds the maximum size of 4 MB.
  Write-Error: /home/runner/work/_temp/36d402fe-2384-47c8-90c9-250096c1bdbc.ps1:70
  Line |
    70 |  Test-TemplateDeployment @functionInput -Verbose
       |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       | Template is not valid.
  Error: Error: The process '/usr/bin/pwsh' failed with exit code 1

job-logs.txt

Manual deployment using Bicep build -f main.test.bicep and az deployment sub create works and deploys the test correctly.

[kpantos]

@Azure/avm-core-team-technical-bicep any updates on this issue?

[jtracey93]

Hey @k pantos, can you share a link to the fork/branch where you are working on and what file this is failing for please?

@ReneHezser would you be able to investigate once we get the details shared above?

[kpantos]

Hey @jtracey93, you can see the issue here https://github.com/kpantos/bicep-registry-modules/actions/runs/12521716173/job/34929149890

[ReneHezser]

I see the compiled tests are 3927 kB in size (which is close to the maximum ARM templates support). Let me check the pipeline.

[kpantos]

That's what I thought initially, but running the test manually through command line succeeds.

[ReneHezser]

I've been looking at the CI and see that the test-deployment returns with the error about the 4MB limit.

Write-Information "Starting test-deployment"
$DeploymentInputs = @{
    "TemplateFile"="C:\Code\kpantos\bicep-registry-modules\avm\ptn\aca-lza\hosting-environment\tests\e2e\waf-aligned\main.test.bicep";
    "DeploymentName"="a-p-al-he-waf-aligned-20250107T1601239878Z";
    "OutVariable"="ValidationErrors";
    "Verbose"=$true;
}
$DeploymentMetadataLocation="germanywestcentral"
# uses https://learn.microsoft.com/en-us/powershell/module/az.resources/test-azdeployment?view=azps-13.0.0
$res = Test-AzSubscriptionDeployment @DeploymentInputs -Location $DeploymentMetadataLocation
if ($res.Message) { Write-Warning $res.Message }

Write-Information "Starting manual deployment"
az deployment sub create --name $DeploymentInputs.DeploymentName --location $DeploymentMetadataLocation --template-file $DeploymentInputs.TemplateFile --what-if

The first part is what is done in the pipeline (or when you test locally via Test-ModuleLocally) and returns with WARNING: The request content size exceeds the maximum size of 4 MB..

I don't see how we can get around this without removing the testing .
But I am open for suggestions :-)

[kpantos]

I'm wondering what is different from doing Bicep build -f main.test.bicep and az deployment sub create -f .\tests\e2e\defaults\main.test.bicep -l northeurope which is working and deploys the test correctly.

Is the CI/CD deployment pipeline leveraging my linked templates https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/linked-templates?tabs=azure-powershell? or is it trying to validate one single arm template generated by bicep buiild?

[jtracey93]

Is it all the tests or just the WAF aligned one as @ReneHezser showed in his testing?

Also has anyone tried/tested building this on a Linux device as that's what the runners will do? Wondering if something to do with case and line endings handling may be in play here.

Regardless getting that close to the 4mb limit is a recipe for future issues anyway. Say I the bicep PG add something to the language in an upcoming release that bloats templates even slightly, then this will cause the issue to reoccur.

Only long term solution in our control is to advise maybe splitting this into multiple pattern modules to shrink the size a bit

[kpantos]

It's all the tests that are failing.

I understand that the avm module needs to be <4MB as it needs to be compiled and stored into the registry (which is by the way a serious limitation for the program as it restricts the composing of more complex solutions). Even splitting into multiple pattern modules, it would defeat the LZA pattern's purpose if these can't be composed together to build a larger pattern but that's another discussion for another time since in this case the module is below that threshold.

The problem probably arises when the tests are also compiled into a single json file together with the module - which for testing it doesn't have to be so. Linked templates can be used for testing which would avoid going over the limit.

[jtracey93]

Hey @kpantos,

Thanks for clarifying 😃

Just to clarify something around the 4MB limit, this isn't an AVM only thing or a container registry thing for storing modules, it's a hard limit in ARM itself for any one template. And because bicep compiles everything into a single ARM template (JSON file), due to the way it currently is designed and engineered, the 4MB limit can be hit more often in bicep than it did in ARM. As you can see lots of long running discussion on the issue over on the Bicep repository for this (the correct place to vent about the 4MB limit 😁 ) Azure/bicep#5890

@ReneHezser lets get this one on the list for this weeks AVM Bicep core team call and see what we can do, if anything.

Finally, @kpantos, whilst the template today is just under the limit, what about in the future, being that close means that we could just be kicking the can down the road and the 4MB limit issue will come back for "real" next time you want to add something to the module. Is there any optimisations and reductions you can make to any of the files used to compose the module, e.g. trimming descriptions, removing whitespace, removing duplicated code etc.? (I actually had to do this for an ALZ module, outside of AVM, as we ran into the 4MB limit 😢 )