Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot prevent az cli from overriding specified version of az cli extension using 'az config set extension.dynamic_install=false' #30615

Open
juliaweingart opened this issue Jan 6, 2025 · 5 comments
Open

Cannot prevent az cli from overriding specified version of az cli extension using 'az config set extension.dynamic_install=false' #30615

juliaweingart opened this issue Jan 6, 2025 · 5 comments
Assignees
@AllyW
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. Configure az configure/config customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning az ml Service Attention This issue is responsible by Azure service team.
Milestone
Backlog

Comments

@juliaweingart
Copy link

Describe the bug

I need to force set the "az ml" extension version to be (2.30.1). I am running these commands:
az extension add --name ml --version 2.30.1
az --version
az ml online-endpoint show --subscription 697c65eb-bf7c-4a92-a66f-6e9e5418ddd0 --resource-group infinity-models --workspace-name infinity-models --name infinite-reasoning-stable

On line 2 when the az version is printed, I see, as expected, that the az ml extension is version 2.30.1.

However, when I run line 3, I am seeing this:
WARNING: Extension 'ml' 2.30.1 is already installed.
WARNING: It will be overridden with version 2.32.4.

It seems the default behavior is to update installed extensions to their latest versions regardless of which version is already installed. In my case, the latest version is broken, so our script is failing. I see online that I can set these flags to stop this default behavior and actually use the pre-installed extension version rather than updating:
az config set extension.dynamic_install=false
az config set extension.dynamic_install_allow_preview=false

These flags aren't working as expected. I still see "az ml" trying to override version 2.30.1 with version 2.32.4.

How do I fix this?

Related command

az config set extension.dynamic_install=false
az config set extension.dynamic_install_allow_preview=false
az extension add --name ml --version 2.30.1
az ml online-endpoint show
--subscription 697c65eb-bf7c-4a92-a66f-6e9e5418ddd0 --resource-group infinity-models --workspace-name infinity-models
--name infinite-reasoning-stable

Errors

No error, but bug that the "az config" commands don't do what they should be doing in making the "az ml" command us the version of "az ml" I explicitly installed.

Issue script & Debug output

az config set extension.dynamic_install=false --debug
DEBUG: cli.knack.cli: Command arguments: ['config', 'set', 'extension.dynamic_install=false', '--debug']
DEBUG: cli.knack.cli: init debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f4a095db7e0>, <function OutputProducer.on_global_arguments at 0x7f4a0937e660>, <function CLIQuery.on_global_arguments at 0x7f4a093c3ba0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Command index version or cloud profile is invalid or doesn't match the current command.
DEBUG: cli.azure.cli.core: Command index has been invalidated.
DEBUG: cli.azure.cli.core: No module found from index for '['config', 'set', 'extension.dynamic_install=false', '--debug']'
DEBUG: cli.azure.cli.core: Loading all modules and extensions
DEBUG: cli.azure.cli.core: Discovered command modules: ['acr', 'acs', 'advisor', 'ams', 'apim', 'appconfig', 'appservice', 'aro', 'backup', 'batch', 'batchai', 'billing', 'botservice', 'cdn', 'cloud', 'cognitiveservices', 'compute_recommender', 'computefleet', 'config', 'configure', 'consumption', 'container', 'containerapp', 'cosmosdb', 'databoxedge', 'dls', 'dms', 'eventgrid', 'eventhubs', 'extension', 'feedback', 'find', 'hdinsight', 'identity', 'interactive', 'iot', 'keyvault', 'lab', 'managedservices', 'maps', 'marketplaceordering', 'monitor', 'mysql', 'netappfiles', 'network', 'policyinsights', 'privatedns', 'profile', 'rdbms', 'redis', 'relay', 'resource', 'role', 'search', 'security', 'servicebus', 'serviceconnector', 'servicefabric', 'signalr', 'sql', 'sqlvm', 'storage', 'synapse', 'util', 'vm']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands
DEBUG: cli.azure.cli.core: acr 0.608 36 149
DEBUG: cli.azure.cli.core: acs 0.284 14 77
DEBUG: cli.azure.cli.core: advisor 0.010 3 6
DEBUG: cli.azure.cli.core: ams 0.012 22 100
DEBUG: cli.azure.cli.core: apim 0.021 14 69
DEBUG: cli.azure.cli.core: appconfig 0.015 9 47
DEBUG: cli.azure.cli.core: appservice 0.145 79 270
DEBUG: cli.azure.cli.core: aro 0.141 1 10
DEBUG: cli.azure.cli.core: backup 0.018 16 60
DEBUG: cli.azure.cli.core: batch 0.408 34 102
DEBUG: cli.azure.cli.core: batchai 0.011 10 30
DEBUG: cli.azure.cli.core: billing 0.056 19 53
DEBUG: cli.azure.cli.core: botservice 0.015 12 42
DEBUG: cli.azure.cli.core: cdn 0.580 8 50
DEBUG: cli.azure.cli.core: cloud 0.010 1 7
DEBUG: cli.azure.cli.core: cognitiveservices 0.006 10 33
DEBUG: cli.azure.cli.core: compute_recommender 0.011 1 1
DEBUG: cli.azure.cli.core: computefleet 0.039 1 1
DEBUG: cli.azure.cli.core: config 0.006 2 7
DEBUG: cli.azure.cli.core: configure 0.005 2 5
DEBUG: cli.azure.cli.core: consumption 0.062 8 9
DEBUG: cli.azure.cli.core: container 0.095 2 17
DEBUG: cli.azure.cli.core: containerapp 0.839 37 123
DEBUG: cli.azure.cli.core: cosmosdb 0.055 58 199
DEBUG: cli.azure.cli.core: databoxedge 0.035 5 28
DEBUG: cli.azure.cli.core: dls 0.006 6 36
DEBUG: cli.azure.cli.core: dms 0.006 3 22
DEBUG: cli.azure.cli.core: eventgrid 0.027 25 96
DEBUG: cli.azure.cli.core: eventhubs 0.078 13 20
DEBUG: cli.azure.cli.core: extension 0.005 1 7
DEBUG: cli.azure.cli.core: feedback 0.005 1 2
DEBUG: cli.azure.cli.core: find 0.005 1 1
DEBUG: cli.azure.cli.core: hdinsight 0.042 9 42
DEBUG: cli.azure.cli.core: identity 0.007 2 11
DEBUG: cli.azure.cli.core: interactive 0.005 1 1
DEBUG: cli.azure.cli.core: iot 0.867 19 82
DEBUG: cli.azure.cli.core: keyvault 0.032 20 113
DEBUG: cli.azure.cli.core: lab 0.203 9 29
DEBUG: cli.azure.cli.core: managedservices 0.064 3 8
DEBUG: cli.azure.cli.core: maps 0.007 5 13
DEBUG: cli.azure.cli.core: marketplaceordering 0.016 1 2
DEBUG: cli.azure.cli.core: monitor 3.312 18 61
DEBUG: cli.azure.cli.core: mysql 0.576 16 56
DEBUG: cli.azure.cli.core: netappfiles 0.199 8 17
DEBUG: cli.azure.cli.core: network 0.875 103 338
DEBUG: cli.azure.cli.core: policyinsights 0.041 9 17
DEBUG: cli.azure.cli.core: privatedns 0.066 14 60
DEBUG: cli.azure.cli.core: profile 0.006 2 8
DEBUG: cli.azure.cli.core: rdbms 0.484 50 208
DEBUG: cli.azure.cli.core: redis 0.011 7 38
DEBUG: cli.azure.cli.core: relay 0.197 7 8
DEBUG: cli.azure.cli.core: resource 0.063 52 232
DEBUG: cli.azure.cli.core: role 0.007 17 61
DEBUG: cli.azure.cli.core: search 0.027 7 19
DEBUG: cli.azure.cli.core: security 0.041 47 98
DEBUG: cli.azure.cli.core: servicebus 0.080 13 16
DEBUG: cli.azure.cli.core: serviceconnector 0.131 20 315
DEBUG: cli.azure.cli.core: servicefabric 0.177 27 80
DEBUG: cli.azure.cli.core: signalr 0.012 10 42
DEBUG: cli.azure.cli.core: sql 0.055 56 215
DEBUG: cli.azure.cli.core: sqlvm 0.322 4 20
DEBUG: cli.azure.cli.core: storage 0.101 59 273
DEBUG: cli.azure.cli.core: synapse 0.078 54 246
DEBUG: cli.azure.cli.core: util 0.007 3 7
DEBUG: cli.azure.cli.core: vm 0.347 58 233
DEBUG: cli.azure.cli.core: Total (65) 12.081 1184 4648
DEBUG: cli.azure.cli.core: Loading extensions:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands Directory
DEBUG: cli.azure.cli.core: azure-devops 0.453 60 192 /opt/az/azcliextensions/azure-devops
DEBUG: cli.azure.cli.core: Total (1) 0.453 60 192
DEBUG: cli.azure.cli.core: Loaded 1230 groups, 4840 commands.
DEBUG: cli.azure.cli.core: Updated command index in 0.003 seconds.
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f4a0861ec00>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/runner/.azure/commands/2025-01-06.23-04-04.config_set.2771.log'.
INFO: az_command_data_logger: command args: config set {} --debug
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7f4a08226d40>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7f4a08226de0>, <function register_cache_arguments..add_cache_arguments at 0x7f4a08226f20>, <function register_upcoming_breaking_change_info..update_breaking_change_info at 0x7f4a08226fc0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [<function _documentdb_deprecate at 0x7f4a06bca520>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f4a0937e700>, <function CLIQuery.handle_query_parameter at 0x7f4a093c3c40>, <function register_ids_argument..parse_ids_arguments at 0x7f4a08226e80>, <function DevCommandsLoader.post_parse_args at 0x7f4a027fe160>]
WARNING: Command group 'config' is experimental and under development. Reference and support levels: https://aka.ms/CLI_refstatus
DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x7f4a08657c40>, <function _x509_from_base64_to_hex_transform at 0x7f4a08657ce0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f4a0861ee80>]
INFO: az_command_data_logger: exit code: 0
INFO: cli.main: Command ran in 15.206 seconds (init: 2.352, invoke: 12.854)
DEBUG: cli.azure.cli.core._profile: Failed to load installationId from AzureRmSurvey.json. [Errno 2] No such file or directory: '/home/runner/.azure/AzureRmContextSettings.json'
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3532 in cache file under /home/runner/.azure/telemetry/20250106230405260
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.12/site-packages/azure/cli/telemetry/init.py /home/runner/.azure /home/runner/.azure/telemetry/20250106230405260"
INFO: telemetry.process: Return from creating process 2781
INFO: telemetry.main: Finish creating telemetry upload process.

  • az config set extension.dynamic_install_allow_preview=false --debug
    DEBUG: cli.knack.cli: Command arguments: ['config', 'set', 'extension.dynamic_install_allow_preview=false', '--debug']
    DEBUG: cli.knack.cli: init debug log:
    Cannot enable color.
    DEBUG: cli.knack.cli: Event: Cli.PreExecute []
    DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f4433edb7e0>, <function OutputProducer.on_global_arguments at 0x7f4433c4e660>, <function CLIQuery.on_global_arguments at 0x7f4433c93ba0>]
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
    DEBUG: cli.azure.cli.core: Modules found from index for 'config': ['azure.cli.command_modules.config']
    DEBUG: cli.azure.cli.core: Loading command modules:
    DEBUG: cli.azure.cli.core: Name Load Time Groups Commands
    DEBUG: cli.azure.cli.core: config 0.002 2 7
    DEBUG: cli.azure.cli.core: Total (1) 0.002 2 7
    DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
    DEBUG: cli.azure.cli.core: Loading extensions:
    DEBUG: cli.azure.cli.core: Name Load Time Groups Commands Directory
    DEBUG: cli.azure.cli.core: Total (0) 0.000 0 0
    DEBUG: cli.azure.cli.core: Loaded 2 groups, 7 commands.
    DEBUG: cli.azure.cli.core: Found a match in the command table.
    DEBUG: cli.azure.cli.core: Raw command : config set extension.dynamic_install_allow_preview=false
    DEBUG: cli.azure.cli.core: Command table: config set
    DEBUG: cli.azure.cli.core: remaining : extension.dynamic_install_allow_preview=false
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f4432ee6c00>]
    DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/runner/.azure/commands/2025-01-06.23-04-05.config_set.2785.log'.
    INFO: az_command_data_logger: command args: config set {} --debug
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7f4432f1fec0>]
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7f4432f4e2a0>, <function register_cache_arguments..add_cache_arguments at 0x7f4432f4e3e0>, <function register_upcoming_breaking_change_info..update_breaking_change_info at 0x7f4432f4e480>]
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f4433c4e700>, <function CLIQuery.handle_query_parameter at 0x7f4433c93c40>, <function register_ids_argument..parse_ids_arguments at 0x7f4432f4e340>]
    WARNING: Command group 'config' is experimental and under development. Reference and support levels: https://aka.ms/CLI_refstatus
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x7f4432f1fc40>, <function _x509_from_base64_to_hex_transform at 0x7f4432f1fce0>]
    DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
    DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
    DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f4432ee6e80>]
    INFO: az_command_data_logger: exit code: 0
    INFO: cli.main: Command ran in 0.128 seconds (init: 0.117, invoke: 0.011)
    INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
    INFO: telemetry.client: Accumulated 0 events. Flush the clients.
    INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
    INFO: telemetry.save: Save telemetry record of length 3545 in cache file under /home/runner/.azure/telemetry/20250106230405818
    INFO: telemetry.main: Begin creating telemetry upload process.
    INFO: telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.12/site-packages/azure/cli/telemetry/init.py /home/runner/.azure /home/runner/.azure/telemetry/20250106230405818"
    INFO: telemetry.process: Return from creating process 2790
    INFO: telemetry.main: Finish creating telemetry upload process.

Expected behavior

I wanted az ml command to use the version I installed, not update the version to the latest before executing.

Environment Summary

az --version
azure-cli 2.67.0
core 2.67.0
telemetry 1.1.0
Extensions:
azure-devops 1.0.1
ml 2.30.1
Dependencies:
msal 1.31.0
azure-mgmt-resource 23.1.1
Python location '/opt/az/bin/python3'
Extensions directory '/opt/az/azcliextensions'
Python (Linux) 3.12.7 (main, Nov 13 2024, 04:02:11) [GCC 11.4.0]

Additional context

No response
@juliaweingart juliaweingart added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Jan 6, 2025
@microsoft-github-policy-service microsoft-github-policy-service bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. Auto-Assign Auto assign by bot labels Jan 6, 2025
@yonzhan
Copy link
Collaborator

yonzhan commented Jan 6, 2025

Thank you for opening this issue, we will look into it.

@microsoft-github-policy-service microsoft-github-policy-service bot added Service Attention This issue is responsible by Azure service team. Machine Learning az ml Configure az configure/config labels Jan 6, 2025
@microsoft-github-policy-service microsoft-github-policy-service bot added Azure CLI Team The command of the issue is owned by Azure CLI team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jan 6, 2025
@yogeshg yogeshg mentioned this issue Jan 6, 2025
Closed
@yonzhan yonzhan removed Azure CLI Team The command of the issue is owned by Azure CLI team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jan 6, 2025
@jsntcy jsntcy assigned AllyW and unassigned jiasli Jan 7, 2025
@AllyW
Copy link
Member

AllyW commented Jan 7, 2025 •
edited
Loading

@juliaweingart please also provide the debug log for running cmd az extension add --name ml --version 2.30.1 --debug and az ml online-endpoint show --subscription 697c65eb-bf7c-4a92-a66f-6e9e5418ddd0 --resource-group infinity-models --workspace-name infinity-models --name infinite-reasoning-stable --debug?

Didn't catch a clue of error from the log of running az config set extension.dynamic_install_allow_preview=false and az config set extension.dynamic_install=false.

If possible, please also provide the result of running pip list for issue debugging, Thanks.

@yonzhan yonzhan added this to the Backlog milestone Jan 7, 2025
@juliaweingart
Copy link
Author

sure I can @AllyW. There are three problems here I believe:

  1. Most recent version of az ml is broken (2.32.4) due to this issue Importing MLClient leads to error "ImportError: cannot import name '_T' from 'marshmallow.fields' " azure-sdk-for-python#39047
  2. When we run the below commands, the version used in "az ml" command is not 2.30.1, but instead it is being auto-upgraded to 2.32.4. This may be expected behavior but seems odd to me and we want a way to force set az ml version in case of such issues where latest versions are broken causing our scripts to break (this is the second time).
    az extension add --name ml --version 2.30.1
    az ml online-endpoint show
  3. I thought that the below command would "solve" our issue and force az ml commands to actually use 2.30.1 rather than auto-upgrading to the latest one, but even adding these lines didn't help.
    az config set extension.dynamic_install_allow_preview=false
    az config set extension.dynamic_install=false

@juliaweingart
Copy link
Author

This solution from this github issue fixed problems 1 and 2 described in my last comment!
az extension add -n ml
pip install marshmallow==3.23.2 --target /opt/az/azcliextensions/ml --upgrade

However, please help with problem 3 in case this issue arises again! How can I force az ml to use a specific version of az ml without upgrading?? @AllyW

@AllyW
Copy link
Member

AllyW commented Jan 8, 2025 •
edited
Loading

@juliaweingart after installed marshmallow 3.23.2 as you posted, the target cmd az ml online-endpoint show --subscription 697c65eb-bf7c-4a92-a66f-6e9e5418ddd0 --resource-group infinity-models --workspace-name infinity-models --name infinite-reasoning-stable --debug works properly in our env. It didn't raise the version overriding warning, just tells me the subscription 697c65eb-bf7c-4a92-a66f-6e9e5418ddd0 cannot be recognized (which is expected because this subscription is yours).

Image

Please provide the debug log for problematic cmd az extension add --name ml --version 2.30.1 --debug and az ml online-endpoint show --subscription 697c65eb-bf7c-4a92-a66f-6e9e5418ddd0 --resource-group infinity-models --workspace-name infinity-models --name infinite-reasoning-stable --debug in your env and we can figure out what's happening when executing this cmd.

BTW, you can also check whether ml is actually working properly when you run az ml -h --debug. If possible, you can paste the log here. I doubt the issue from ml extension makes the dynamic install process keeps trying to fetch a new version. Thanks.

The debug log for az config set extension.dynamic_install_allow_preview=false and az config set extension.dynamic_install=false seems ok in our side, and we didn't find ml in it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AllyW AllyW

Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. Configure az configure/config customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning az ml Service Attention This issue is responsible by Azure service team.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@jiasli @AllyW @yonzhan @juliaweingart