[Feature Request] - Run Amazon Linux 2023 as a virtual machine on premises

[Manouchehri]

Is your feature request related to a problem? Please describe.
Creation of Amazon Linux 2022 virtual machine (VM) images for on-premises development and testing. e.g. the same as: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-2-virtual-machine.html

Describe the solution you'd like
This URL to be valid:

https://cdn.amazonlinux.com/os-images/2022/kvm/

Describe alternatives you've considered
I can probably figure out how to make my own bootstrapped install using dnf install --installroot=/mnt, but it would be nice to have an official image to start with.

[chadbrewbaker]

I'm working on this right now! Trying to get a UTM image for my M1, here are the instructions for Arch: https://ktprograms.codeberg.page/blog/posts/2022-03-17_1750_utm-arch-arm/

The first steps I think can be refactored into an Alpine linux container for x86/ARM.

• Build container
• Image for wsl2 - upstream to https://github.com/yosukes-dev/AmazonWSL
• Image for UTM - upstream to https://github.com/utmapp/mac.getutm.app/tree/main/_virtual_machines
• Image for kvm

[Manouchehri]

@chadbrewbaker Hmm, not sure I get how that helps with AL2022?

[liesenml]

Hello, thank you for submitting your request! We review all requests on a weekly basis and will reply to this ticket by with our evaluation.

[chadbrewbaker]

Bump. This is nontrivial. I blew another day on it yesterday - need some official AWS help here.

My notes for inspecting Fedora images on macos:

strings Fedora-Minimal-36-1.5.armhfp.raw | grep RSA "PRIVATE"
xxd -a Fedora-Minimal-36-1.5.armhfp.raw | less
fdisk Fedora-Minimal-36-1.5.armhfp.raw
hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount Fedora-Minimal-36-1.5.armhfp.raw
hdiutil mount /dev/disk5s1

From what I can tell AmazonLinux2 used effectively the same grub2 VM structure as Fedora. https://arm.fedoraproject.org

@iximiuz has project to take the OSI container and stuff a bootloader etc on it: iximiuz/docker-to-linux#25 This might be the most elegant solution.

[bplessis-swi]

The FAQ still references kvm images as to be provided on the GA release:

Q: Can I use AL2023 outside of AWS?

A: AL2023 images can be used outside of AWS, however, these images are not covered by AWS Support Plans when used outside of AWS. When Amazon Linux 2023 becomes Generally Available, it will be provided as a virtual machine image for on-premises use, enabling you to easily develop, test, and certify applications from a local development environment. This option is not available during the preview.

Look like someone forgot this ^^

[nmeyerhans]

Indeed the FAQ did indicate that VM images would be available at GA for use outside of EC2. Let's consider that FAQ entry to be inaccurate, for now.

We are working on VM images and will note their availability in the release notes and resolve this issue when they're available.

[emboss64]

Any updates on this one? Thanks!

[yoonjs2]

Any updates? Thanks!

[r-vw]

Also very interested in any progress wrt this request

[Rosch]

I am missing AL2023 in a vmware instance desperately! Is there anything known when this comes to life?

[rotan-hanrahan]

According to the Wayback Machine, the following text was present on https://aws.amazon.com/linux/amazon-linux-2023/faqs/ in March 2023:

When Amazon Linux 2023 becomes Generally Available, it will be provided as a virtual machine image for on-premises use, enabling you to easily develop, test, and certify applications from a local development environment. This option is not available during the preview.

The June 3 snapshot of the same page no longer contains this text, so the promise made during the preview regarding the VM image once AL2023 reaches GA appears to be withdrawn. (Note: the revision of the FAQ was suggested by @nmeyerhans in early April, almost three months ago.)

I see no evidence of any outreach to the community to explain this, nor any requests for technical assistance (assuming the issue is technical). If the issue is bureaucratic in nature, we might never see the promised VM image.

Some clarification from Amazon is overdue.

[inflatador]

Red Hat's recent announcement opens a window of opportunity for Amazon Linux to greatly increase on-prem adoption. Here's hoping Amazon will take advantage of that opportunity, starting with the VM images requested here.

[tanvp112]

It's becoming more and more difficult to use AL2 now, eg. https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html#common-upgrading-issues

[stewartsmith]

Apologies for the long time radio-silence here, I understand the frustration this must cause, and there's certainly developers like myself who share that frustration.

While this isn't an image, it may be a useful starting point for building your own. Using the pretty neat kiwi-ng with the following config, I can boot to a login prompt. There's a few other things that are missing that turn this into a base image that we'd release (some cloud-init config at least), but it's a starting point.

You'll need this at root/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200n8 console=tty0 rd.emergency=poweroff rd.shell=0 selinux=1 security=selinux quiet"
GRUB_TIMEOUT=0
GRUB_DISABLE_RECOVERY="true"
GRUB_TERMINAL="ec2-console"
GRUB_ENABLE_BLSCFG="true"
GRUB_X86_USE_32BIT="true"
GRUB_DEFAULT=saved
GRUB_UPDATE_DEFAULT_KERNEL=true

and this at root/etc/cloud/cloud.cfg.d/02-onprem.cfg

datasource_list: [ NoCloud, AltCloud, ConfigDrive, OVF, None ]

and then your appliance.kiwi can look like:

<?xml version="1.0" encoding="utf-8"?>

<image schemaversion="7.5" name="amazonlinux-2023">
    <description type="system">
        <author>Stewart Smith</author>
        <contact>[email protected]</contact>
        <specification>Example Amazon Linux 2023 kiwi config</specification>
    </description>
    <profiles>
        <profile name="Virtual" description="Virtual image of Amazon Linux 2023"/>
    </profiles>
    <preferences>
        <version>2.0.0</version>
        <packagemanager>dnf</packagemanager>
        <keytable>us</keytable>
        <timezone>UTC</timezone>
        <rpm-check-signatures>true</rpm-check-signatures>
    </preferences>
    <preferences profiles="Virtual">
      <type image="oem" filesystem="xfs" kernelcmdline="console=ttyS0" firmware="efi" format="qcow2">
            <oemconfig>
                <oem-resize>false</oem-resize>
            </oemconfig>
            <bootloader name="grub2" console="serial" timeout="10"/>
        </type>
    </preferences>
    <repository imageinclude="false" sourcetype="mirrorlist">
      <source path="https://cdn.amazonlinux.com/al2023/core/mirrors/latest/x86_64/mirror.list"/>
    </repository>
    <packages type="bootstrap" patternType="onlyRequired">
      <package name="dnf"/>
      <package name="curl-minimal"/>
      <package name="gnupg2-minimal"/>
    </packages>
    <packages type="image" patternType="onlyRequired">
      <namedCollection name="onprem"/>
      <namedCollection name="onprem-minimal"/>
      <package name="kernel"/>
      <package name="grub2-pc"/>
      <package name="grub2-pc-modules"/>
      <package name="grub2-efi-x64-modules"/>
      <package name="grub2-efi-x64-ec2"/>
      <package name="zstd"/>
      <package name="systemd-networkd"/>
    </packages>
</image>

Built on a Fedora 38 instance, and tested with qemu-system-x86_64 -boot c -net nic -net user -m 4G -cpu max -smp 4 -hda testboot -bios /usr/share/edk2/ovmf/OVMF_CODE.fd -nographic just to booting to a Login prompt, and not a moment further.

i.e. this is a starting point for someone to continue what that looks like.

[chadbrewbaker]

Apparently you can do this with AWS CLI? https://docs.aws.amazon.com/vm-import/latest/userguide/vmexport.html

al2023-ami is the keyword apparently?

[cpswan]

I've been keeping an eye on this issue for a while, and it seems AWS have really dropped the ball here.

Given the chaos in the Enterprise Linux (EL) space since RedHat announced changes to RHEL source availability there was a real opportunity for Amazon Linux to become the preferred distro for those wanting big company backing without subscribing to RHEL (or dealing with Oracle).

But that opportunity is being squandered, on this very issue. If people can't run AL23 everywhere that they want EL, then they'll probably choose something else as their baseline.

Furthermore, promising images and then failing to deliver makes it look like the project isn't properly resourced, which is another ding on trust.

Choice of distro has been a topic I've been following for a long time (since well before this post almost 7y ago). It seems almost incomprehensible that AWS isn't grabbing hold of the chance to get customers onto AL wherever they run Linux, which of course smoothes the path to migrating stuff to AWS.

Repo maintainers... this isn't aimed at you. I guess you're struggling to keep everything together with insufficient resources. This is for you to show to the higher ups, so they're making an informed decision about where they're placing their bets, and the consequences of ruined expectations and inaction.

[AaronSwearingen]

What OS does AWS recommend for hybrid cloud solutions? AL2 is less then two years away from EOL. AL2023 isn't available as a virtual machine and also presumably isn't supported for use with https://aws.amazon.com/hybrid/ features like ECS/EKS Anywhere.

I appreciate the repo maintainers trying to help out the community, but AWS needs to be made aware this is more than just a community request. There is a real business need to enable companies to leverage (and pay for) AWS hybrid cloud services. Anyone expecting AL2023 to fulfill that need now finds themselves in a position of uncertainty.

[chadbrewbaker]

What OS does AWS recommend for hybrid cloud solutions? AL2 is less then two years away from EOL. AL2023 isn't available as a virtual machine and also presumably isn't supported for use with https://aws.amazon.com/hybrid/ features like ECS/EKS Anywhere.

I appreciate the repo maintainers trying to help out the community, but AWS needs to be made aware this is more than just a community request. There is a real business need to enable companies to leverage (and pay for) AWS hybrid cloud services. Anyone expecting AL2023 to fulfill that need now finds themselves in a position of uncertainty.

Usual suspects - Ubuntu, Debian, Fedora, Suse, Arch. ML practitioners love Ubuntu, Linus loves Fedora.

I would probably use Fedora Desktop as it is closest to AL2023 for laptops.

This Reddit thread is spot on. AWS lost its way firing passionate maintainers like Tim McNamara that would have knocked out this ticket a year ago to get more community test coverage/adoption.

[AaronSwearingen]

I would probably use Fedora Desktop as it is closest to AL2023 for laptops.

ECS Anywhere supports Fedora 33 according to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html. That's the latest version of Fedora on this list - EOL since 2021-11-30. The rest of the list isn't looking much healthier.

Getting AL2023 on the list would go a long way towards AWS showing it is committed to supporting hybrid cloud.

[hesaurey]

Amazon please release OVA for Vmware it's good for you, good for us!

[cpswan]

The Register referenced this issue with their piece Amazon Linux 2023 virtual machine images still MIA

It closes with the usual:

Amazon did not respond to a request for comment

But let's hope that it helps get some attention onto fixing the problem.

[Rosch]

Hi Amazon

An on premise version of AL 2023 is truly missing. When will it be available?
We want to fully support AL2023 with our software but developing for it
is only possible when it is available in our intranet. So yet we can only support AL2.

Beside of helping software developers an on premise version of AL 2023
is really a good chance for AL to fill the gap caused by RHELs actions.

Thanks

Roland

[danie-dejager]

I installed a host with AL2023 on-prem using DistroBox. It's adequate for my own limited use case, not the same as a VM running AL2023 though.

[ozbenh]

We found the cause of the issue and will be updating the images. A sad story involving different versions of open-vmdk having different (incompatible) syntax for mkova.sh ... ugh. It should have been caught, sorry about that, something slipped in the test validation. Stay tuned for updated images. We'll also be slimming them a bit as mentioned earlier by @stewartsmith

[ozbenh]

A fixed (and slimmed) OVA has been uploaded and replaces the previous one. We haven't changed the KVM ones at all for now

[ccassidy-maxar]

A fixed (and slimmed) OVA has been uploaded and replaces the previous one. We haven't changed the KVM ones at all for now

Tested the new OVA and worked flawlessly on VMWare Workstation 17.0.2 / Windows 10, thanks!

[tanvp112]

I'm bound to a VirtualBox environment, and have used the VB images from Amazon ( such as here https://cdn.amazonlinux.com/os-images/2.0.20231101.0/ ), so I guess I will wait while the bugs are ironed out of the kvm/vmware versions and see what happens. So glad to see movement on this again, though I did lose one client to the "dark side" on account of no offsite pre-deployment options like we had with the previous images. (I might try to pull them back if the VB image appears.)

I'll take a quick peek once the URLs stop returning 404s.

Thanks @stewartsmith.

The OVA can be imported into VB 6.1.48, but after change password there seems to be storage driver incompatibility; either the new password is not recognized upon reboot or error with "Authentication token manipulation". Will need to try 7.x and possibly tune the seed files...

[ozbenh]

I'm bound to a VirtualBox environment, and have used the VB images from Amazon ( such as here https://cdn.amazonlinux.com/os-images/2.0.20231101.0/ ), so I guess I will wait while the bugs are ironed out of the kvm/vmware versions and see what happens. So glad to see movement on this again, though I did lose one client to the "dark side" on account of no offsite pre-deployment options like we had with the previous images. (I might try to pull them back if the VB image appears.)
I'll take a quick peek once the URLs stop returning 404s.
Thanks @stewartsmith.

The OVA can be imported into VB 6.1.48, but after change password there seems to be storage driver incompatibility; either the new password is not recognized upon reboot or error with "Authentication token manipulation". Will need to try 7.x and possibly tune the seed files...

VirtualBox has been .... weird ... can you try converting the image away from vmdk to vdi (I suppose there's a way to do that, I haven't researched) first ? I would not be surprised if VB has bugs around vmdk handling...

[tanvp112]

The VMDK formatted hard drive has no issue convert into VDI. Previously for AL2, a VB image is provided by AWS and it works out of the box. Believe it or not, I have run this copy & still running today for >5 years day-in-day-out. AL2 is arguably the best Linux experience I ever had so far.

The release of AL2023 image is a big step to many... kudos for AWS Linux team. Hopefully, there will be a VB image like the former and the superb experience would continue. For the case above, VB 6.1.x is sunsetting already, will need to test with latest VB 7.x instead.

Happy to hear more about others' experience using VB with the new image!

[yoonjs2]

Is possible to convert AL2023 arm64 qcow2 to parallels hdd format? I'm bounded to macOS with apple silicon environment and uses parallels in dev/production.

[ozbenh]

Is possible to convert AL2023 arm64 qcow2 to parallels hdd format? I'm bounded to macOS with apple silicon environment and uses parallels in dev/production.

I don't have a Mac to try (though @stewartsmith does, he might come up with an answer), but Google has a few hits on this. Alternatively there's a qemu for Mac which uses Hypervisor Framework under the hood and there's UTM as a front-end for it all

[ozbenh]

Is possible to convert AL2023 arm64 qcow2 to parallels hdd format? I'm bounded to macOS with apple silicon environment and uses parallels in dev/production.

I don't have a Mac to try (though @stewartsmith does, he might come up with an answer), but Google has a few hits on this. Alternatively there's a qemu for Mac which uses Hypervisor Framework under the hood and there's UTM as a front-end for it all

I did some experimentations and it didn't go particularly well. Not only converting to parallel is tricky and messy (the tools don't work well and parallels itself doesn't seem to be helping), but the end result doesn't boot.

I think I know why (they and vmware fusion both seem to be exposing storage as some kind of ACPI generated platform AHCI that isn't PCI backed, and we don't enable that driver). I'll see if I can get that turned on in a future kernel release. That said, we don't have plans at this stage to produce officially supported Parallels images.

[michailf]

Does anyone have a reference to login information? What user and password would I use for the images downloaded from the links provided?

[azN2]

Does anyone have a reference to login information? What user and password would I use for the images downloaded from the links provided?

Cloud-init

[stewartsmith]

Does anyone have a reference to login information? What user and password would I use for the images downloaded from the links provided?

Be sure to read the instructions on https://docs.aws.amazon.com/linux/al2023/ug/outside-ec2.html for configuring the VM. There isn't a password set, you need to set up something so that cloud-init can configure the instance on first boot.

[michailf]

I spent hours trying to figure out how to login to the provided image. I have read multiple documents, including the reference here. I cannot log in to the image. I was able to set the local hostname via the meta-data (guestinfo.metadata) so I believe I provide the data correctly. What I do not know is why the users are not being setup with a simple password.

Is there a working example of the user-data (guestinfo.userdata) that allows a simple password log in with sudo rights to the image provided?

[michailf]

I was able to log in using an ssh key. I then could create/adjust other users to my needs. It seems the cloud-init in the image does not allow for plain passwords.

[tanvp112]

I think I know why (they and vmware fusion both seem to be exposing storage as some kind of ACPI generated platform AHCI that isn't PCI backed, and we don't enable that driver).

This really rings a bell! I suspect this is required for the image to be compatible for VB as well.

[gebailey]

The VMDK formatted hard drive has no issue convert into VDI. Previously for AL2, a VB image is provided by AWS and it works out of the box.

I've built Vagrant cloud images (virtualbox provider) of Amazon Linux 2 for a few years now, and just uploaded one for Amazon Linux 2023 that was built from the VMware OVA. A few steps were required to convert from OVA->VMDK->VDI but seems to work just fine. https://app.vagrantup.com/gbailey/boxes/al2023

[ozbenh]

I was able to log in using an ssh key. I then could create/adjust other users to my needs. It seems the cloud-init in the image does not allow for plain passwords.

I have used images with passwords a few times but usually by creating a new user, something like:

  - name: benh
    groups: sudo
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    ssh-authorized-keys:
      - <key>
    plain_text_passwd: <password>
    lock_passwd: false

This doesn't seem to work with ec2-user last I tried, but did you check if the old-school chpasswd method works ? From the AL2 on-prem docs:

chpasswd:
  list: |
    ec2-user:<plain_text_password>

[michailf]

I have used images with passwords a few times but usually by creating a new user, something like:

  - name: benh
    groups: sudo
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    ssh-authorized-keys:
      - <key>
    plain_text_passwd: <password>
    lock_passwd: false

I got ssh-authorized-keys to work but not plain_text_passwd independently. I have not tried both of the options for the same user.

This doesn't seem to work with ec2-user last I tried, but did you check if the old-school chpasswd method works ? From the AL2 on-prem docs:

chpasswd:
  list: |
    ec2-user:<plain_text_password>

I also tried the chpasswd method but it did not work for me.

[ozbenh]

I just tested again with

#cloud-config
#vim:syntax=yaml
users:
  - name: test
    groups: sudo
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    plain_text_passwd: passw0rd
    lock_passwd: false

and it worked fine

Are you logging in via the console or serial ? Or are you trying to ssh ? Passsword authentication is disabled in the ssh server by default.

[michailf]

I just tested again with

#cloud-config
#vim:syntax=yaml
users:
  - name: test
    groups: sudo
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    plain_text_passwd: passw0rd
    lock_passwd: false

and it worked fine

I tried your example and it did work. I am not sure what I was doing wrong before.

The following example works to create a new user (al) with sudo access, enable the default user (ec2-user), set the ec2-user's password and force a password change on login.

#cloud-config
#vim:syntax=yaml
users:
  - default
  - name: al
    groups: sudo
    sudo: ALL=(ALL) NOPASSWD:ALL
    plain_text_passwd: Passwor8
    lock_passwd: false
chpasswd:
  list: |
    ec2-user:Passwor8

The following is a minimal example to gain full access to the system as ec2-user has sudo rights:

#cloud-config
#vim:syntax=yaml
users:
  - default
chpasswd:
  list: |
    ec2-user:Passwor8

[denisgabriel5]

Does anyone have some instructions on how to modify the VMware image according to this article?

[nileshtrivedi]

Is there a tutorial somewhere for running AL2023 on Apple M1 machines (perhaps via UTM)?

[ozbenh]

I did run for fun and no profit our qcow2 image on an M1 mac, I think using UTM a while back, though I don't remember the details. It's not a tested/supported solution so we don't have official docs, but I can try to snatch my partner's M1 one of these days and try again & let you know

[nileshtrivedi]

@ozbenh Actually I got it working with some effort. This is the user-data file that went into seed.iso attached as a CDROM in the UTM machine:

#cloud-config
#vim:syntax=yaml
password: test123
chpasswd:
  expire: False

users:
  - default
  - name: ec2-user
    lock_passwd: false
    plain_text_passwd: test123
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-ed25519 yourkey_goes_here

It help to delete the .qcow2 and .iso disks fully and add again after every change. I think this is because most cloud-init config takes effect only on "first boot".

[yoonjs2]

For people looking for the al2023 vagrant box + apple silicon (arm64) + parallels desktop, I created and uploaded working box here: https://portal.cloud.hashicorp.com/services/vagrant/registries/hbsmith/boxes/al2023

I created the box using conversions on the mac like this: qemu qcow2 -> vmware vmdk -> parallels hdd then pack it into pvm & vagrant box. AL2023 has no problem with installing parallels tools extension and work with vagrant like a charm. (except manually assign fixed ip to network interface.)

Hope it helps.

[ozbenh]

Ah I was about to update here but you beat me to it :-) Yes, our ARM64 image seems to work with UTM and the default "virt" machine type (I think it's 7.2) on an M1. As for seed.iso , well ... that's documented in various places :-)

The one thing that I noticed doesn't work is Secure Boot as you can't enroll our keys, the problem seems to be that UTM comes up pre-provisioned with its own set of keys instead of in Setup Mode. It might be worth filing a feature request for that. We don't currently provide a pre-made UEFI variables blob in qemu format but you can use python3-virt-firmware or python-uefivars to convert ours from AWS format to Qemu format (you can find our blob and our keys in /usr/share/amazon-linux-sb-keys/, the UEFI variables blob is uefi.vars.

It would be nice if UTM provided AL2023 in their Gallery I suppose ...