Implement Ray for distributed task execution

[ake2l]

Overview

Introduces Ray framework for distributed task execution and multiprocessing support in Datamimic. This change moves us from single-process execution to a distributed model where:

• Parent process handles model parsing and orchestration
• Ray workers handle parallel data generation and processing
• Improved resource utilization across multiple cores

Key Features

• New GenerateWorker class using Ray for distributed execution
• Parent/worker process separation for better resource management
• Automatic scaling across available CPU cores
• Proper context isolation between workers
• Efficient cleanup of resources

Technical Implementation

• Ray workers handle chunks of data generation tasks
• Parent process manages task distribution and result aggregation
• Context copying ensures proper isolation between workers
• Resource cleanup in finally blocks
• Error propagation from workers to parent process

Architecture

Parent Process

• Parse models
• Initialize Ray
• Distribute tasks to workers
• Aggregate results

Ray Workers (@ray.remote)

• Handle data generation
• Process data chunks
• Manage database connections
• Clean up resources

Notes

This is our first implementation using Ray for distributed processing. The architecture separates concerns between parent orchestration and worker execution while maintaining our existing model parsing and validation.

[ake2l]

@dangsg a first draft of what we discussed , you can have a look ... all unittest and nearly all integration test are already working successfully