Release - version 1.1.0

.gitignore

@@ -1,2 +1,2 @@
 *.DS_Store
-local
+.env-shared.sh

.gitmodules

@@ -1,7 +1,3 @@
-[submodule "binhosts/ps3-gentoo-binhosts"]
-	path = binhosts/ps3-gentoo-binhosts
-	url = [email protected]:damiandudycz/ps3-gentoo-binhosts.git
-	branch = main
 [submodule "overlays/ps3-gentoo-overlay"]
 	path = overlays/ps3-gentoo-overlay
 	url = [email protected]:damiandudycz/ps3-gentoo-overlay.git
@@ -10,7 +6,7 @@
 	path = overlays/ps3-gentoo-overlay.distfiles
 	url = [email protected]:damiandudycz/ps3-gentoo-overlay.distfiles.git
 	branch = main
-[submodule "releases/ps3-gentoo-stages"]
-	path = releases/ps3-gentoo-stages
-	url = [email protected]:damiandudycz/ps3-gentoo-stages.git
+[submodule "releases"]
+	path = releases
+	url = [email protected]:damiandudycz/ps3-gentoo-releases
 	branch = main

README.md

@@ -1,100 +1,61 @@
 # PlayStation 3 Gentoo Linux Toolset
 
-Welcome to the PlayStation 3 Gentoo Linux Toolset repository — a comprehensive collection of files and tools designed for seamless installation and maintenance of Gentoo Linux on the PlayStation 3. Here's an overview of the current features:
-
-- Automatic Installer: Facilitates the effortless installation and configuration of Gentoo Linux on the PS3.
-- Prebuilt Kernel: Compatible with both OtherOS and OtherOS++, this kernel is equipped with various patches specifically tailored for the PS3.
-- Binhost Repository: Streamlines the installation process by enabling quicker access to diverse packages without the need for direct compilation.
-- Portage overlay: Overlay repository featuring additional tools specifically crafted for the PS3.
-- Developer Tools: Essential for building new kernel versions and managing different aspects of the repository.
-- LiveDVD/LiveUSB Image: An immersive option allowing you to run Gentoo directly from a USB device.
-
-The repository undergoes constant updates, incorporating new features and addressing any issues. Future plans include:
-
-- Customized Petitboot: Developing an enhanced and up-to-date version of Petitboot for an optimized user experience.
-- Installer Integration with Petitboot: Making adjustments to the installer to seamlessly integrate with Petitboot for user-friendly functionality.
-- Enhanced Prebuilt Kernel: Continuously improving the prebuilt kernel to ensure optimal performance.
-- Automatic Binrepo Update Tool: Implementing an automatic tool to keep the binhost repository up-to-date effortlessly.
-
-I value your feedback and encourage you to share your comments or suggestions using the "Issues" tab or via email at [email protected].
-Your input is invaluable, and all comments are greatly appreciated! :)
-
-For development puropses clone with: `git clone --recurse-submodules [email protected]:damiandudycz/ps3.git`
-To download installer use: `wget https://raw.githubusercontent.com/damiandudycz/ps3/main/installer/ps3-gentoo-installer`
-
-# Installer
-
-Installer is located at `insteller/installer.sh`. You can use it from another linux to install Gentoo on the PS3 hard drive or in selected directory.
-You can use it with default configuration or you can adjust the settings to match your needs.
-TODO: Add more information about the installer.
-
-### OLD description
-
-Im constantly working on making this project more reliable and eaisy to use. Recently I added a binhost repository which is automatically added to Gentoo during installation, to make the process faster. Currently testing it. 
-Please feel free to report any issues you have or some suggestions/questions using Issues tab in GitHub.
-
-If you want to try Gentoo using LiveUSB / LiveDVD, download files from https://www.dropbox.com/scl/fi/qrjzexfm1owd1r1aprkd8/LiveUSB.zip?rlkey=kd3isyeo5tkhqtbb1sy57hbh6&dl=1
-and extract them to USB formatted with MBR / FAT32
-
-# Kenrel building tools
-
-These tools should be used on a host machine, running `Gentoo`. You can use `Virtual Machine` for that puropse.
-
-After cloning the repository, go to downloaded `ps3/tools` directory and edit hidden `.config` file
-Select the version of kernel you want to work on, and make sure the version you selected is available in portage. Also in this file you can select the `address of the PS3` in your local network.
-
-These tools will apply specific kernel patches from two seperate sources. If you want to modify the list of applied patches, edit the file `01-setup-patches.sh` and change `patches_t2sde` and `patches_ps3linux_patches`. 
-
-After that you can run the script: `01-setup-patches.sh`. It will download all selected patches, to be used for kernel later.
-
-Next run `02-setup-sources.sh`, this will download `gentoo-sources`, and apply previously prepared patches.
-
-Run `03-kernel-configure.sh`, and customize configuration to your needs. Please use the script instead of running menuconfig directly, as the scripts adds special crossdev layer.
-When you are sacisfied with your config, run `04-kernel-build.sh`.
-
-At this point kernel is ready, and you can find it in `/var/cache/ps3tools/linux`
-
-If you want you can upload it directly to the `PS3` using `05-kernel-upload-to-ps3.sh`. This required the `root access` for the SSH to be configured for the PS3. If you don't want to use `root`, you can edit `05-kernel-upload-to-ps3.sh` and change root to another user, but it needs the write access to `/boot`. Otherwise you can copy it manually from `/var/cache/ps3tools/linux`. Also make sure `/boot` is mounted on your PS3 before running it.
-
-Currently there is no tool for automatic creation of `kboot/yaboot` files, so if you changed the kernel version, you need to add these entries manually.
-
-# Gentoo-Installer
-
-Automatic installer and configurator of Gentoo linux for various platforms.
-
-To install on the PS3, boot into any recent linux distribution, setup the date and networking and:
-
-To install on the whole drive:
-
-`./ps3-gentoo-installer --device /dev/ps3dd --config PS3 --verbose`
-
-this will format selected harddrive!
-
-To install into selected directory without formatting the drive:
-
-`./ps3-gentoo-installer --directory /mnt/gentoo --config PS3 --verbose`
-
-and after installer finished, add fstab configuration and kboot entry.
-
-If you want to customize configuration, you can download file config/ps3, edit it and use as
-
-`./ps3-gentoo-installer --device /dev/ps3dd --custom-config ps3_file_path --verbose`
-
-To use distcc during installation, use --distcc flag:
-
-`./ps3-gentoo-installer --device /dev/ps3dd --config PS3 --distcc "192.168.0.50"`
-
-# Acknowledgement
-
-If you use my project as a base for other forks or projects, please add contribution with url to my repository.
-
-Special thanks to:
-- Model Citizen PS3 - for sharing knowledgle about Linux on PS3,
-- Rene Rebe - for updating and maintaining kernel patches for the PS3,
-- Immolo - for sharing knowledge about Gentoo,
-- OtherOS++ team - for making it all possible.
-
-
-
-
-
+Welcome to the PlayStation 3 Gentoo Linux Toolset repository—a comprehensive collection of files and tools designed to facilitate the seamless installation and maintenance of Gentoo Linux on the PlayStation 3.
+
+## Repository Contents
+
+### Minimal Installation CD
+- **install-cell-minimal.iso**: This ISO file can be burned to a CD/DVD or a USB drive and launched on the PS3 via the Petitboot menu. It supports both automatic and manual Gentoo installations. For automatic installation, use the `ps3-gentoo-installer` program included on the CD. Note that Petitboot must be installed on your PS3 to boot from this ISO.
+
+### PS3 Gentoo Installer
+- **ps3-gentoo-installer**: This tool facilitates the automatic installation and configuration of Gentoo Linux using the Minimal Installation CD. It performs the following tasks:
+    - Formats and partitions the hard drive for Gentoo.
+    - Downloads and extracts Stage3.
+    - Synchronizes the portage tree.
+    - Installs `gentoo-kernel-ps3`.
+    - Configures Petitboot.
+    - Installs additional tools.
+    - Sets up a default user with sudo access.
+    - And more…
+
+    After execution, Gentoo can be booted on the PS3. Note that installation can take several hours, depending on the packages to be compiled. **Warning:** This tool will format your hard drive unless you choose the directory installation method. Please back up any sensitive data beforehand.
+
+    To install, run:
+        ps3-gentoo-installer --device /dev/ps3dd
+
+    For more options, use:
+        ps3-gentoo-installer --help
+
+### Releases
+- **releases/23.0-default directory**: Contains the following:
+    - Minimal install CD ISO files for booting on the PS3 using Petitboot.
+    - Stage3 files built for the CELL CPU, offering better compatibility with the PS3 compared to the default PPC64 Stage3 files.
+    - Stage4 files, with xorg and mate installed.
+- **releases/ppc directory**: A collection of binhost repositories that can be added to your portage configuration. These repositories are generated during the release process of Autobuilds and contain pre-compiled packages optimized for the CELL CPU. If you use the `ps3-gentoo-installer`, these repositories will be added automatically. Recommended repository URL:
+    - https://raw.githubusercontent.com/damiandudycz/ps3-gentoo-releases/main/ppc/binpackages/23.0/cell
+
+### Overlays
+- **Overlays/ps3-gentoo-overlay**: A portage overlay containing ebuilds for packages useful on the PS3 system, including:
+    - `gentoo-kernel-ps3`: A modified gentoo-kernel package with additional patches and configurations for the PS3, which also adds a Petitboot entry.
+    - `gentoo-sources-ps3`: A modified gentoo-sources package with additional patches and configurations for the PS3.
+    - `ps3vram-swap`: An RC script that utilizes PS3 VRAM as a swap device and configures system parameters for better memory management on the PS3.
+    - `ps3-gentoo-installer`: The automatic Gentoo installer for the PS3, available on the Minimal Installation CD.
+
+    If you use `ps3-gentoo-installer`, this overlay will be added automatically. To manually add `ps3-gentoo-overlay` to your portage configuration, run:
+        eselect repository add ps3 git https://github.com/damiandudycz/ps3-gentoo-overlay
+
+### Development Tools
+- **dev-tools directory**: Contains tools used by the repository developer for maintenance and helper tasks. These include:
+    - `update-submodules.sh`: Initializes submodule repositories and githooks after cloning this repository.
+    - `distcc-docker`: Configuration for a Docker image to set up a DistCC server, aiding the PS3 in compiling code faster.
+    - `kernel-ebuild-builder`: Manages and updates `ps3-gentoo-overlay` ebuild files related to the kernel.
+    - `ps3-gentoo-installer`: The current version of the automatic Gentoo installer for the PS3, available on the Minimal Installation CD.
+    - `release-builder`: Creates and uploads new releases, including:
+        - Minimal Installation CD
+        - Stage3 files
+        - Binhost repositories
+
+## Feedback and Contributions
+If you have any suggestions or encounter any issues, please feel free to contact me at [[email protected]](mailto:[email protected]) or find me on the Gentoo Discord @damiandudycz.
+
+If you use any of my tools in your project, please include a reference to this repository and a link back to it. Your acknowledgment helps support the development and maintenance of these tools.

dev-tools/binhost/README.md

@@ -0,0 +1,2 @@
+The directory contains scripts used to maintain binhost repository.
+For generating new packages, use scripts from dev-tools/release group.

dev-tools/binhost/binhost-01-delete-packages.sh

@@ -0,0 +1,115 @@
+#!/bin/bash
+
+# This script deletes a specified package from the binhost repository.
+# Usage example: ./binhost-delete-package.sh sys-kernel/gentoo-kernel-ps3.
+# It can also remove packages larger than specified size - this functionality
+# can be used with or without specifying package name.
+
+source ../../.env-shared.sh || exit 1
+register_usage "$0 --pkgcache <PKG_CACHE_DIRECTORY> <package>[-version] --if-larger <SIZE_LIMIT>"
+
+# Parse input parameters
+declare -a ARG_PACKAGES_TO_REMOVE
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --verbose) ;; # Handled by env-shared.sh
+        --pkgcache|-p)
+            PKGCACHE_DIR="$2"
+            shift 2
+            ;;
+        --if-larger|-s)
+            SIZE_LIMIT=$2
+            SIZE_LIMIT_ARG=$2
+            case "${SIZE_LIMIT: -1}" in
+                K|k) SIZE_LIMIT=$(( ${SIZE_LIMIT%K*} * 1024 )) ;;
+                M|m) SIZE_LIMIT=$(( ${SIZE_LIMIT%M*} * 1024 * 1024 )) ;;
+                G|g) SIZE_LIMIT=$(( ${SIZE_LIMIT%G*} * 1024 * 1024 * 1024 )) ;;
+            esac
+            shift 2
+            ;;
+        --*|-*)
+            failure "Unknown parameter: $1"
+            ;;
+        *)
+            # If the package doesn't contain a category, prepend '*' to match any category
+            if [[ "$1" != */* ]]; then
+                ARG_PACKAGES_TO_REMOVE+=("*/$1")
+            else
+                ARG_PACKAGES_TO_REMOVE+=("$1")
+            fi
+            shift
+            ;;
+    esac
+done
+
+# Check if package parameter is provided and package exists
+[[ ! -z "${PKGCACHE_DIR}" ]] || show_usage
+[[ ! -z "${ARG_PACKAGES_TO_REMOVE}" ]] || [ ${SIZE_LIMIT} ] || show_usage
+
+# Convert wildcard to regex
+convert_to_regex() {
+    local WILDCARD="$1"
+    WILDCARD="${WILDCARD//\*/.*}"
+    WILDCARD="${WILDCARD//\?/.}"
+    echo "^${WILDCARD}(-[0-9][0-9.a-zA-Z]*)?(_p[0-9]+)?(-r[0-9]+)?$"
+}
+
+# Convert package names with wildcards to regex
+declare -a ARG_PACKAGES_TO_REMOVE_REGEX
+for PACKAGE in "${ARG_PACKAGES_TO_REMOVE[@]}"; do
+    ARG_PACKAGES_TO_REMOVE_REGEX+=("$(convert_to_regex "$PACKAGE")")
+done
+
+# Process metadata file
+PKGCACHE_METADATA="${PKGCACHE_DIR}/Packages"
+PACKAGES_COUNT=$(grep -oP '^PACKAGES: \K[0-9]+' "${PKGCACHE_METADATA}")
+VAR_METADATA_NEW=""
+ENTRY=""
+unset ENTRY_DELETE
+unset METADATA_MODIFIED
+while IFS= read -r LINE || [[ -n $LINE ]]; do
+    if [[ -z "${LINE}" ]]; then
+        ENTRY_PACKAGE=$(awk -F'/' '{print $1"/"$2}' <<< "${ENTRY_PATH}")
+        ENTRY_VERSION="${ENTRY_CPV#"$ENTRY_PACKAGE-"}"
+        ENTRY_PACKAGE_VERSIONED="${ENTRY_PACKAGE}-${ENTRY_VERSION}"
+        for PACKAGE_REGEX in "${ARG_PACKAGES_TO_REMOVE_REGEX[@]}"; do
+            if [[ ! ${SIZE_LIMIT} ]] || [[ ${ENTRY_SIZE} -gt ${SIZE_LIMIT} ]]; then
+                if [[ ! -z "${ENTRY_CPV}" ]] && [[ "${ENTRY_PACKAGE_VERSIONED}" =~ ${PACKAGE_REGEX} ]]; then
+                    ENTRY_DELETE=true
+                fi
+            fi
+        done
+        if [[ ! -z "${ENTRY_CPV}" ]] && [[ -z "${ARG_PACKAGES_TO_REMOVE_REGEX}" ]] && [[ ${ENTRY_SIZE} -gt ${SIZE_LIMIT} ]]; then
+            ENTRY_DELETE=true
+        fi
+        if [[ -n ${ENTRY_DELETE} ]]; then
+            echo "Removing ${ENTRY_CPV} [${ENTRY_PATH}]"
+            ((PACKAGES_COUNT--)) || PACKAGES_COUNT=0
+            METADATA_MODIFIED=true
+            rm -f "${PKGCACHE_DIR}/${ENTRY_PATH}"
+        else
+            VAR_METADATA_NEW+="${ENTRY}\n"
+        fi
+        ENTRY=""; ENTRY_CPV=""; ENTRY_PATH=""; ENTRY_SIZE=0; unset ENTRY_DELETE
+    else
+        ENTRY+="${LINE}\n"
+        KEY=${LINE%% *}
+        if [[ "${KEY}" == "CPV:" ]]; then
+            ENTRY_CPV=${LINE#* }
+        elif [[ "${KEY}" == "PATH:" ]]; then
+            ENTRY_PATH=${LINE#* }
+        elif [[ "${KEY}" == "SIZE:" ]]; then
+            ENTRY_SIZE=${LINE#* }
+        fi
+    fi
+done < "${PKGCACHE_METADATA}"
+
+# Save changes
+if [ ${METADATA_MODIFIED} ]; then
+    echo -e "${VAR_METADATA_NEW}" > "${PKGCACHE_METADATA}"
+    sed -i "s/^PACKAGES: .*/PACKAGES: $PACKAGES_COUNT/" "${PKGCACHE_METADATA}"
+elif [ ${SIZE_LIMIT} ]; then
+    echo "Packages: ${ARG_PACKAGES_TO_REMOVE[@]} larger than ${SIZE_LIMIT_ARG} not found in repository."
+else
+    echo "Packages: ${ARG_PACKAGES_TO_REMOVE[@]} not found in repository."
+fi

dev-tools/binhost/binhost-02-sanitize.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# This function removes packages larger than size limit in github.
+
+source ../../.env-shared.sh || exit 1
+register_usage "$0 --pkgcache|-p <PKG_CACHE_DIRECTORY>"
+
+# Parse input parameters
+declare -a ARG_PACKAGES_TO_REMOVE
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --pkgcache|-p)
+            PKGCACHE_DIR_TO_SANITIZE="$2"
+            shift 2
+            ;;
+	*)
+            show_usage
+            ;;
+    esac
+done
+
+[[ ! -z "${PKGCACHE_DIR_TO_SANITIZE}" ]] || show_usage
+
+source ${PATH_BINHOST_SCRIPT_DELETE} --pkgcache "${PKGCACHE_DIR_TO_SANITIZE}" --if-larger ${CONF_GIT_FILE_SIZE_LIMIT}