Enterprise Level AI Application Architecture Request: Feature Request / Project Proposal

[shravan-ks-donyati]

Project Overview: Company Internal AI Chatbot (Semantic Kernel-based)

Project Description
This project aims to develop an AI chatbot similar to ChatGPT but tailored for internal company use. The chatbot will provide context-aware AI responses by integrating Semantic Kernel and various Azure services to achieve efficient, real-time, and dynamic responses. The system will support features like document retrieval, Bing Search, streaming responses, and Graph API operations (e.g., sending emails or messages).

Core Features

• Context-Aware Responses

1. The chatbot retrieves and responds based on the most relevant past conversations.

• Document-Based RAG (Retrieval-Augmented Generation)

1. Embeddings of textual files or images will be stored in Azure AI Search Vector Store.
2. The system should fetch the relevant document chunks based on user queries and generate accurate AI responses.

• Bing Search Integration

1. When no relevant data is found in the vector store, the system should fall back to using the Bing Search API for up-to-date information.

• Streaming Responses

1. The system must always stream AI responses to the client, rather than providing a single-shot response.

• Graph API Integration

1. The chatbot should use the Microsoft Graph API to perform operations such as sending emails or messages.

Questions for Implementation

• Using Semantic Kernel for Features

1. How can I integrate Semantic Kernel to manage context-aware responses, document retrieval, Bing Search, and Graph API operations seamlessly?

• Performance Optimization

1. How can I ensure the response time remains low, especially when integrating multiple services (Azure AI Search, Bing Search, and Graph API)?

• Best Azure Services

1. Which Azure services are best suited for developing this AI system? (e.g., Azure OpenAI, Azure Functions, Azure Cosmos DB, Azure AI Search)

• Kernel Integration

1. Can I integrate all plugins (native plugins, prompt template plugins, and external API integrations) into a single kernel, or do I need to use multiple kernels?

• Planners

1. Should I use a single planner to orchestrate different tasks (e.g., search, graph operations) or multiple planners for different functionalities?

• Streaming Responses

1. Does the response from the planner support streaming responses? If not, what modifications or strategies can be used to achieve streaming responses?

• Dynamic Plugin Selection

1. The system should dynamically decide among multiple plugins (e.g., Azure Search, Bing Search, Graph API) based on the context of the user's query.
2. The response should always be streamed back to the client.

Proposed Architecture
Frontend

• Framework: React.js
• Purpose: User interaction, streaming response rendering

Backend (Azure Functions)

• FastAPI for handling RAG requests and AI responses
• Azure Functions for CRUD operations, Graph API calls, and indexing workflows

Semantic Kernel

• Memory Plugins: Azure Cognitive Search for vector retrieval
• Bing Search Plugin: For internet-based fallbacks
• Graph API Plugin: For interacting with Microsoft Graph operations

Azure Services

• Azure OpenAI: GPT-4o model for responses, ada-002 for embeddings
• Azure AI Search: Storing and retrieving document embeddings
• Azure Blob Storage: Storing uploaded files (images, documents)
• Azure Cosmos DB: Storing chat and message metadata
• Application Insights: For monitoring and logging

Additional Context
The AI system should efficiently manage embeddings, fallbacks, and streaming responses to provide a seamless experience.
Future enhancements may include collaboration features, search filtering, and data visualization capabilities.

Request for Feedback
I would appreciate insights and suggestions on:

1. Best practices for implementing these features using Semantic Kernel.
2. Recommendations for Azure service integration.
3. Strategies for optimizing response time and performance.
4. Whether single or multiple kernels and planners are ideal for this project.
5. How to achieve streaming responses with planners.
6. the ai system should decide

Note: The backend and Azure Functions are implemented using Python, and the client/frontend is built with React.js. I am not using C# or .NET in this project.

Thank you for your guidance!