[Feature] adapt fused sigmoid gate for MoE model

[zhyncs]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

ref https://github.com/NVIDIA/TensorRT-LLM/blob/be1788106245496872d18e702978e59b6bfd50e0/cpp/tensorrt_llm/kernels/mixtureOfExperts/moe_kernels.cu#L232

Related resources

No response

[zhaochenyang20]

@NovTi

[sitabulaixizawaluduo]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

ref https://github.com/NVIDIA/TensorRT-LLM/blob/be1788106245496872d18e702978e59b6bfd50e0/cpp/tensorrt_llm/kernels/mixtureOfExperts/moe_kernels.cu#L232

Related resources

No response

LMDeploy should also do the same optimization, which should greatly improve the performance of MoE