Appearance
Java SDK, Qwen-Omni, real-time, multimodal
Reference, synced 2026-06-13.
flowchart TD n0["Models"] n1["Overview"] n2["Products"] n3["Solutions"] n4["Pricing"] n5["Resources"] n6["Partners"] n7["Support"] n8["Language"] n0 --> n1 n1 --> n2 n2 --> n3 n3 --> n4 n4 --> n5 n5 --> n6 n6 --> n7 n7 --> n8
Free access Accelerate Delivery with Fixed-Cost Agentic CodingWatch how it works
Models
Empowering AI innovation for both enterprises and developers with Alibaba Cloud’s best-in-class Qwen models, AI-native apps, and AI solutions.
Alibaba Cloud Model Studio \ Enterprise-grade large model service and application development platform.
Try Visual Model \ Supports image understanding, image generation, and video generation.
Models
HappyHorse-1.0-T2V \ Cinematic creative generation, ultimate dynamic details Qwen3-VL-Plus \ Native VL, spatial reasoning, 1M-context video analysis Wan2.7-VideoEdit \ Supports both localized and global editing with prompt
Qwen3.6-Plus \ Native multimodal, 1M context, agentic coding Wan2.7-Image-Pro \ Interactive editing, long-text rendering, precise prompt following Qwen-Plus \ Balanced intelligence, efficient inference, production-ready performance
Qwen-Image-2.0 \ Professional infographics, exquisite photorealism Z-Image-Turbo \ Ultra-fast image generation, high throughput, cost-optimized inference Qwen3-Coder-Next \ Multi-turn tool interactions, future-ready development support
Wan2.7-T2V \ High-fidelity T2V, 15s duration, advanced camera control Wan2.7-I2V \ Cinematic I2V with emotional depth and visceral impact Wan2.7-R2V \ Up to 5 mixed image/video inputs and audio timbre cloning
GenAI Application
Qoder \ Intelligent coding assistant, available for enterprise-dedicated deployment. Qoder CN \ AI-powered coding assistant that boosts developer productivity with intelligent code completion, AI chat, multi-file editing, and task automation.
AI Service
Model Experience \ Experience full-scale, multimodal model capabilities online. Platform for AI \ An AI-native algorithm engineering platform for end-to-end modeling, training, and inference service deployment. Fine-tune Video Generation Model \ Customize Wan’s text-to-video capabilities through model fine-tuning to meet your unique requirements.
AI Use Case
AI Savings Plan Hot \ Save up to 47% on AI costs. Limited-time offer tailored to your usage. AI Video Creation \ Elevate your professional video production with Wan 2.6.
AI Token Plan \ One plan. Multiple models. Big Savings with a Fixed Subscription. AI Image Creation \ All-in-one creative suite for copywriting, image generation, and poster design.
Overview
As a global full-stack AI leader, Alibaba Cloud aims to make computing accessible to everyone and help worldwide customers accelerate innovation.
Why Alibaba Cloud
About Alibaba Cloud \ AI Powered Cloud Technology Our Global Network \ Explore our global presence and deployment regions around the world Our Global Offices \ With offices in 4 continents, we're always close to where it matters.
Asia Accelerator \ Accelerate Success in Asia with Alibaba Cloud Go Global \ Benefits of our Global Alliance Trust Center \ Empowering enterprises with a secure, compliant, and globally trusted cloud infrastructure
Customers and Insights
Olympic Games \ Alibaba Cloud Powers Olympic Games with AI-powered cloud technology Case Studies \ Learn how customers are scaling their businesses on Alibaba Cloud Analyst Reports \ Learn what the top industry analyst firms are saying about Alibaba Cloud
What's New
Events and Webinars \ Quick access to upcoming and on-demand events Product Updates \ Stay informed of the latest innovations Press Room \ Latest news and media releases
Products
Featured ProductsAI & Machine Learning Computing Container Storage Networking & CDN Security Middleware Database Analytics ComputingMedia ServicesEnterprise Services & Cloud CommunicationDomain Names and WebsitesEnd User ComputingServerlessDeveloper ToolsMigration & O&M ManagementApsara Stack
Featured Products
Alibaba Cloud Model Studio \ Supercharge your AI journey effortlessly with industry-leading GenAI models ApsaraDB RDS \ Store and manage your business data, with automated monitoring and backups Certificate Management Service (Original SSL Certificate) \ Create a safe and secure connection between your website and users
Elastic Compute Service (ECS) \ Host websites anywhere and scale enterprise workloads Container Service for Kubernetes (ACK) \ Run and scale containerized applications on managed Kubernetes infrastructure Object Storage Service (OSS) \ Store large amounts of data in the cloud and access it anywhere, anytime
Simple Application Server (SAS) \ All-in-one services for fast deployment Elastic IP Address (EIP) \ Manage your public IPs independently to improve internet network quality Domain Names and Website \ Get the perfect domain name to suit your every need
Related Programs
Solutions
Solutions by Industry Technical Solutions AI WebsitesNetworking Security and ComplianceData and AnalyticsEnterprise Service and ApplicationCloud MigrationCloud NativeHybrid CloudSMB solutions
Financial Services \ Innovate faster with Alibaba Cloud Games \ Grow your game rapidly with high global availability
New Retail \ Alibaba Cloud enables digital retail transformation to fuel growth and realize an omnichannel customer experience throughout the consumer journey. Media and Entertainment \ Ready your content for today's media market with a digitalized media journey
Supply Chain \ Power your supply chain with intelligent, efficient, and reliable solutions Sports \ Digitizing the sports industry with intelligent tech
Sustainability \ Achieve a sustainable future with low-carbon and energy-efficient technologies
Pricing
Flexible options like pay-as-you-go and clear billing rules to meet diverse business needs.
Overview & Tools
Pricing Calculator \ Get an instant pricing estimate based on your usage and needs Free Trial \ Try our 80+ cloud products for free.
Pricing Options \ Get the most out of Alibaba Cloud with flexible pricing
Optimize your cost
Migrate & Save \ Superior Performance At Lower Pricing. Save up to 50%. Promotion Center \ Unlock the latest Alibaba Cloud offers & promos
Resources
Official documentation, extensive tools, training resources, and a community to grow and innovate in the cloud.
Technical Resources
Documentation \ Product guides and FAQs Architecture Center \ Design reliable, secure, and efficient cloud architecture. Intelligent Solution Explorer \ Find the right solution for you, powered by AI
Blog \ Latest cloud insights and developer trends Whitepapers \ Research that explores the how and why behind our technology
Training&Certification
Alibaba Cloud Academy \ Build cloud skills and earn certifications with expert-led training.
Developer Hub
Alibaba Cloud Project Hub \ Explore real-world projects built by developers using our platform. Our Developer MVPs \ Celebrating the developers who lead, build, and inspire our community
Partners
Partner-first strategy offering collaborative product, sales, and service models, plus high-quality partner solutions that complement Alibaba Cloud’s capabilities.
Marketplace
AI Alliance for ISVs \ Partner with us to build and grow AI solutions together ISV Benefits \ Unlock resources, market access, and go-to-market support as an ISV partner
Alibaba Cloud Marketplace \ Explore ready-to-deploy solutions from our partners and ISVs
Find a Partner
Partner Hub \ Find your ideal partner in no time
Become a Partner
Partner Network \ A partner portal for Alibaba Cloud Channel, Technology, MSP partner and other partner programs
Support
Full-lifecycle support and expert services, from cloud advisory and migration to operations.
Support & Professional Services
Professional Services \ Expert-led services to design, migrate, and optimize your cloud journey Support Plans \ Flexible support for every stage — from startup to enterprise
Partner Support Program \ Priority technical support for partners, with dedicated managers and faster issue resolution
Contact us
Connect With Us \
Talk to a sales expert and get a custom quote for your business
Language
- English
- 简体中文
- 繁體中文
- 日本語
- Bahasa Indonesia
Locale
Visit aliyun.com
Documentation
Alibaba Cloud Model Studio
User Guide (Models) User Guide (Application) API Reference (Models) API Reference (Application)
Search for Help Content
Getting Started
The Beginner's Guide
Well-Architected Framework
AI & Machine Learning
Platform For AI
Alibaba Cloud Model Studio
DashVector
Artificial Intelligence Recommendation
OpenSearch
Image Search
Machine Translation
Intelligent Speech Interaction
Optimization Solver
Intelligent Computing LINGJUN
Computing
Elastic Compute Service
Elastic GPU Service
Elastic Container Instance
Dedicated Host
Compute Nest
Simple Application Server
Cloud Box
Auto Scaling
Elastic High Performance Computing
Batch Compute (Deprecated)
Function Compute
Serverless App Engine
ENS
Elastic Desktop Service
App Streaming
WUYING Terminal
Cloud Phone
Edge Network Acceleration
Alibaba Cloud Linux
AgentBay
Container
Container Service for Kubernetes
Container Compute Service
Container Registry
Storage
Object Storage Service
Cloud Parallel File Storage
File Storage NAS
Tablestore
Storage Capacity Unit
Simple Log Service
Cloud Backup
Intelligent Media Management
Drive and Photo Service
Data Transport
Cloud Storage Gateway
Data Online Migration
Hybrid Cloud Storage Array
Storage Services Overview
Backup and Disaster Recovery Center
Networking and CDN
Server Load Balancer
Elastic IP Address
Internet Shared Bandwidth
Data Transfer Plan
Virtual Private Cloud
NAT Gateway
PrivateLink
Alibaba Cloud DNS PrivateZone
Network Intelligence Service
Cloud Data Transfer
IPv6 Gateway
Anycast Elastic IP Address
Cloud Enterprise Network
Global Accelerator
VPN Gateway
Smart Access Gateway
Express Connect
CDN
Edge Security Acceleration
Cloud Network Well-architected Design Guidelines
Security
Anti-DDoS
Web Application Firewall
Cloud Firewall
Security Center
Bastionhost
Secure Access Service Edge
Certificate Management Service
Key Management Service
Data Security Center
Identity as a Service
Fraud Detection
AI Guardrails
Captcha
Blockchain as a Service
ID Verification
Managed Security Service
Middleware
Enterprise Distributed Application Service
Microservices Engine
Alibaba Cloud Service Mesh
SchedulerX
ApsaraMQ for RocketMQ
ApsaraMQ for Kafka
ApsaraMQ for RabbitMQ
ApsaraMQ for MQTT
Simple Message Queue (formerly MNS)
CloudFlow
EventBridge
Application Real-Time Monitoring Service
Managed Service for Prometheus
Managed Service for Grafana
Managed Service for OpenTelemetry
Performance Testing
STAROps
Databases
ApsaraDB Console
PolarDB
ApsaraDB RDS
ApsaraDB for OceanBase (Deprecated)
Tair (Redis® OSS-Compatible)
Lindorm
Time Series Database
ApsaraDB for MongoDB
ApsaraDB for HBase
ApsaraDB for Memcache
ApsaraDB for MyBase
AnalyticDB
ApsaraDB for ClickHouse
ApsaraDB for SelectDB
Data Transmission Service
Database Autonomy Service
Data Management
Database Gateway - Deprecated
ApsaraDB for Cassandra - Deprecated
Analytics Computing
MaxCompute
Hologres
Realtime Compute for Apache Flink
Elasticsearch
Vector Retrieval Service for Milvus
E-MapReduce
Data Lake Formation
DataV
Quick BI
Quick Audience
Quick Tracking
DataWorks
DataHub
Dataphin
Media Services
ApsaraVideo VOD
ApsaraVideo Live
Intelligent Media Services
ApsaraVideo Media Processing
Apsara Video SDK
Enterprise Services & Cloud Communication
Energy Expert
CloudQuotation
Salesforce on Alibaba Cloud
GoChina ICP Filing Assistant
Marketplace
Alibaba Mail
Direct Mail
Short Message Service
Voice Service
Phone Number Verification Service
Cell Phone Number Service
Chat App Message Service
Financial Intelligence Engine
Domain Names and Websites
Domain Names
ICP Filing
Alibaba Cloud DNS
End User Computing
Elastic Desktop Service
App Streaming
WUYING Terminal
Cloud Phone
AgentBay
Internet of Things
IoT Platform
Serverless
Serverless App Engine
CloudFlow
EventBridge
Simple Message Queue (formerly MNS)
Function Compute
Developer Tools
OpenAPI Explorer
Alibaba Cloud SDK
Cloud Shell
Resource Orchestration Service
Alibaba Cloud CLI
BSS OpenAPI
Terraform
Pulumi
Ticket System API
Mobile Platform as a Service
Alibaba Cloud DevOps
API Gateway
Cloud Control API
AI Coding Assistant Lingma
Cloud Skills Portal
Migration & O&M Management
CloudOps Orchestration Service
Cloud Monitor
Intelligent Advisor
Cloud Governance Center
ActionTrail
Cloud Config
Resource Access Management
Resource Management
Cloud Architect Design Tools
Migration Hub
Server Migration Center
Service Catalog
Logic Composer
Quota Center
CloudSSO
HTTPDNS
Solutions
SAP
SuperApp
OpenLake
Membership Service
Expenses and Costs
Account Center
More
Support
Legal
Tech Share Terms and Conditions
After Sales Support
China Gateway Program
Service Level Objectives
Management Console
Security Control
The key interfaces and request parameters for Qwen-Omni real-time DashScope Java SDK.
Prerequisites
Your Java SDK version must be v2.22.15 or later. Before you begin, see Real-time multimodal interaction flow.
Getting started
Download the sample code from GitHub. Three calling methods are available:
- Audio conversation example: Captures real-time audio from a microphone, enables Voice Activity Detection (VAD) mode, which automatically detects the start and end of speech, and supports voice interruption.
Set the enableTurnDetection parameter to
true.
Use headphones for audio playback to prevent echoes from triggering voice interruption.
- Audio and video conversation example: Captures real-time audio and video from a microphone and camera, enables VAD mode, and supports voice interruption.
Set the enableTurnDetection parameter to
true.
Use headphones for audio playback to prevent echoes from triggering voice interruption.
- Local call: Uses local audio and images as input and enables Manual mode, which lets you manually control the sending pace.
Set the enableTurnDetection parameter to
false.
Request parameters
Configure the following request parameters using the chained methods or setters of the OmniRealtimeParam object, and then pass it to the OmniRealtimeConversation constructor.
| Parameter | Type | Description |
|---|
| Parameter | Type | Description |
| model | String | The Qwen-Omni model to use. See Model list. |
| url | String | The endpoint URL: - Singapore region: wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime. Replace {WorkspaceId} with your actual workspace ID. - Beijing region: wss://dashscope.aliyuncs.com/api-ws/v1/realtime |
Important
The legacy domain wss://dashscope-intl.aliyuncs.com for the Singapore region will be deprecated soon. Please migrate to the new domain wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com as soon as possible.
Configure the following request parameters using the chained methods or setters of the OmniRealtimeConfig object, and then pass it to the updateSession method.
| Parameter | Type | Description |
|---|
| Parameter | Type | Description |
| modalities | List<OmniRealtimeModality> | The model output modality. Set to [OmniRealtimeModality.TEXT] for text only, or [OmniRealtimeModality.TEXT, OmniRealtimeModality.AUDIO] for both audio and text. |
| voice | String | The voice for audio output. For supported voices, see Voice list. Default voice: - Qwen3.5-Omni: "Tina" - Qwen3-Omni-Flash-Realtime: "Cherry" - Qwen-Omni-Turbo-Realtime: "Chelsie" |
| inputAudioFormat | OmniRealtimeAudioFormat | The format of the user's input audio. Currently only supports PCM_16000HZ_MONO_16BIT, which represents a PCM audio stream at a 16 kHz sample rate. |
| outputAudioFormat | OmniRealtimeAudioFormat | The format of the model's output audio. Currently only supports PCM_24000HZ_MONO_16BIT, which represents a PCM audio stream at a 24 kHz sample rate. |
| instructions | String | A system message that sets the goal or role for the model. Example: "You are an AI customer service agent for a five-star hotel. Answer customer questions about room types, facilities, prices, and booking policies accurately and in a friendly manner. Always respond professionally and helpfully. Do not provide unverified information or information outside the scope of the hotel's services." Note Set instructions using the OmniRealtimeConfig instance's parameters method: java conversation.updateSession(OmniRealtimeConfig.builder() .modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT)) .voice("Tina") .enableTurnDetection(true) .enableInputAudioTranscription(true) .parameters(Map.of( "instructions", "You are a personal assistant named Xiaoyun" )) .build() ); |
| smooth_output | Boolean | This parameter is supported only by the Qwen3-Omni-Flash-Realtime series. - true: The model provides conversational responses. - false: The model provides more formal, written-style responses. > However, performance may be suboptimal if the content is difficult to read aloud. - null: The default value. The model automatically chooses between conversational and formal response styles. > Set smooth_output using the OmniRealtimeConfig instance's parameters method. The usage is the same as instructions. |
| enableInputAudioTranscription | Boolean | Enables speech recognition for input audio. |
| InputAudioTranscription | String | Speech recognition model for input audio transcription. The value is always qwen3-asr-flash-realtime. This parameter is not configurable. |
| enableTurnDetection | Boolean | Enables VAD. If disabled, you must manually submit audio to create a response. |
| turnDetectionType | String | The VAD type. Valid values: - server_vad (default): Detects the end of user speech based on acoustic features. - semantic_vad: Detects the end of user speech based on semantic validity. This mode can filter out meaningless speech, such as backchannels and background noise. This option is supported only by the qwen3.5-omni-realtime model. |
| turnDetectionThreshold | Float | VAD detection threshold. Increase in noisy environments and decrease in quiet environments. - The closer the value is to -1, the more likely noise is to be detected as speech. - The closer the value is to 1, the less likely noise is to be detected as speech. Default: 0.5. Valid values: [-1.0, 1.0]. |
| turnDetectionSilenceDurationMs | Integer | Silence duration that indicates end of speech. If exceeded, the model triggers a response. Default: 800. Valid values: [200, 6000]. |
| turnDetectionParam | Map | Additional turn_detection configuration parameters. Currently supports idle_timeout_ms (Integer): the idle timeout in milliseconds. Applies only to qwen3.5-omni-plus-realtime and qwen3.5-omni-flash-realtime models in server_vad mode. After the server finishes audio playback and the user remains silent beyond this duration (no speech.started triggered), the model proactively generates a response to prompt the user to continue the conversation. Valid range: [5000, 30000]. Example: turnDetectionParam(Map.of("idle_timeout_ms", 5000)) |
| enable_search | Boolean | This parameter takes effect only when you use the Qwen3.5-Omni-Realtime model. Enables web search. Default: false. When enabled, the model can search the web to answer real-time questions. > enable_search and search_options parameters must be set using the parameters method of the OmniRealtimeConfig instance. The process is the same for instructions. > Tool calling (tools) and web search (enable_search) are incompatible and cannot be enabled at the same time. |
| search_options | Object | Web search option settings. This parameter takes effect only after enable_search is enabled. You can set enable_source (Boolean) to true to return a list of search result sources. > search_options using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions. |
| tools | List<Map<String, Object>> | This parameter takes effect only when you use the Qwen3.5-Omni-Realtime model. Tool definitions. When provided, the model can call external tools to respond to user questions. If a tool is called, the model does not generate audio and only returns the tool calling parameters. Each tool is a Map that contains the following fields: - type (String, required): The value is fixed to "function". - function (Map, required): The definition of the tool function. This field contains the following subfields: - name (String, required): A custom name for the tool function. We recommend that you use the same name as the function, such as get_current_weather or get_current_time. - description (String, optional): A description of the tool function. The model uses this description to decide whether to call the function. - parameters (Map, optional): Input parameter descriptions. The model uses them to extract parameters. If the tool function does not require input parameters, you do not need to specify this field. This field contains the following subfields: - type (String, required): The value is fixed to "object". - properties (Map, optional): Describes the name, data type, and description of each input parameter. The Key is the parameter name, and the Value is a Map that contains the data type (type) and description (description). - required (List, optional): Specifies which input parameters are required. > search_options using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions. |
| temperature | Float | Sampling temperature that controls content diversity. Higher values produce more diverse content; lower values produce more deterministic content. Valid values: [0, 2). Because both temperature and top_p control content diversity, we recommend that you set only one of them. Default values of temperature: - qwen3.5-omni-realtime series: 0.7 - qwen3-omni-flash-realtime series: 0.9 - qwen-omni-turbo-realtime series: 1.0 > qwen-omni-turbo models do not support modification. > temperature parameter using the parameters method of the OmniRealtimeConfig instance, in the same way you set the instructions parameter. |
| top_p | Float | Probability threshold for nucleus sampling that controls content diversity. Higher values produce more diverse content; lower values produce more deterministic content. Valid values: (0, 1.0]. Because both temperature and top_p control content diversity, we recommend that you set only one of them. Default values of top_p: - qwen3.5-omni-realtime series: 0.8 - qwen3-omni-flash-realtime series: 1.0 - qwen-omni-turbo-realtime series: 0.01 > qwen-omni-turbo models do not support modification. > top_p using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions. |
| top_k | Integer | Candidate set size for sampling. For example, 50 means only the 50 highest-scoring tokens form the candidate set. Larger values increase randomness; smaller values increase determinism. Set to None or a value greater than 100 to disable top_k and use only top_p. The value must be greater than or equal to 0. Default values of top_k: - qwen3.5-omni-realtime series: 20 - qwen3-omni-flash-realtime series: 50 - qwen-omni-turbo-realtime series: 20 > qwen-omni-turbo models do not support modification. > top_k using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions. |
| max_tokens | Integer | Maximum number of tokens to return. > max_tokens setting does not affect the generation process of the model. If the number of tokens generated by the model exceeds max_tokens, the returned content is truncated. Default and maximum values equal the model's maximum output length. See Model list for details. Use max_tokens to limit output length for generating summaries, keywords, controlling costs, or reducing response time. > qwen-omni-turbo models do not support modification. > max_tokens using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions. |
| repetition_penalty | Float | Controls repetition in generated sequences. Higher values reduce repetition. 1.0 means no penalty. Must be greater than 0. Default values of repetition_penalty: - qwen3.5-omni-realtime series: 1.0 - Other models: 1.05 > qwen-omni-turbo models do not support modification. > repetition_penalty using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions. |
| presence_penalty | Float | Controls content repetition in model output. Valid values: [-2.0, 2.0]. Positive values reduce repetition; negative values increase it. Default values of presence_penalty: - qwen3.5-omni-realtime series: 1.5 - Other models: 0.0 Scenarios: Higher values suit scenarios requiring diversity or creativity, such as creative writing or brainstorming. Lower values suit scenarios requiring consistency or technical precision, such as technical documents. > qwen-omni-turbo models do not support modification. > presence_penalty using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions. |
| seed | Integer | Makes generation more deterministic, ensuring consistent results across runs. Passing the same seed with identical parameters produces the same result as much as possible. Valid values: 0 to 231−1. Default value: -1. > qwen-omni-turbo models do not support modification. > seed using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions. |
Key interfaces
OmniRealtimeConversation class
Import with import com.alibaba.dashscope.audio.omni.OmniRealtimeConversation;.
| Method signature | Server response event (sent via callback) | Description |
|---|
| Method signature | Server response event (sent via callback) | Description |
java public void connect() throws NoApiKeyException, InterruptedException | Server events > Session created session.updated > Session configuration updated | Creates a connection to the server. |
java public void updateSession(OmniRealtimeConfig config) | session.updated > Session configuration updated | Updates session configuration. For parameter details, see Request parameters. After connecting, the server returns default session configurations. Call this method immediately after connecting to update settings. When the server receives the session.update event, it validates the parameters. Invalid parameters return an error; otherwise the server updates session configuration. |
java public void appendAudio(String audioBase64) | None | Appends Base64-encoded audio to the cloud input buffer. The buffer is temporary storage that can be written to and committed later. - If "turn_detection" is enabled, the audio buffer is used to detect speech, and the server decides when to commit. - If "turn_detection" is disabled, the client can send up to 15 MiB of audio per event. Streaming smaller blocks makes VAD more responsive. |
java public void appendVideo(String videoBase64) | None | Adds Base64-encoded image data to the cloud video buffer. Accepts local images or real-time video stream captures. Image input limits: - The image format must be JPG or JPEG. The recommended image resolution is 480p or 720p, with a maximum of 1080p. - A single image after Base64 encoding must not exceed 256 KB. We recommend keeping the raw image size below 190 KB before encoding. - The image data must be Base64-encoded. - We recommend that you send images to the server at a frequency of 1 image per second. |
java public void clearAppendedAudio() | input_audio_buffer.cleared > The audio received by the server is cleared | Deletes the audio in the current cloud buffer. |
java public void commit() | input_audio_buffer.committed > The server received the committed audio | Commits audio and video from the cloud buffer. Returns an error if the buffer is empty. - If "turn_detection" is enabled, the client does not need to send this event. The server automatically commits the audio buffer. - If "turn_detection" is disabled, the client must commit the audio buffer to create a user message item. Note: 1. If input_audio_transcription is configured for the session, the system transcribes the audio. 2. Committing the input audio buffer does not create a response from the model. |
java public void createResponse(String instructions, List<OmniRealtimeModality> modalities) | Server events > The server starts to generate a response response.output_item.added > New output content is available in the response Server events > A conversation item is created response.content_part.added > New output content is added to the assistant message item response.audio_transcript.delta > Incrementally generated transcribed text response.audio.delta > Incrementally generated audio from the model response.audio_transcript.done > Text transcription is complete response.audio.done > Audio generation is complete response.content_part.done > Streaming of text or audio content for the assistant message is complete response.output_item.done > Streaming of the entire output item for the assistant message is complete response.done > The response is complete | Instructs the server to create a model response. When the session is configured in "turn_detection" mode, the server automatically creates a model response. |
java public void cancelResponse() | None | Cancels the in-progress response. If no response is available to cancel, the server returns an error. |
java public void createItem(JsonObject item) | None | Sends the conversation.item.create event to the server. In a tool calling scenario, this method is used to send the tool execution result back to the server. The item parameter is a JsonObject and must contain the following fields: - type: The value is fixed to "function_call_output". - call_id: Corresponds to the call_id in the response.function_call_arguments.done event. - output: A string that represents the tool execution result. |
java public void close() | None | Stops the task and closes the connection. |
java public String getSessionId() | None | Gets the session ID of the current task. |
java public String getResponseId() | None | Gets the response ID of the most recent response. |
Callback interface (OmniRealtimeCallback)
The server returns response events and data via callbacks. Implement callback methods to process server responses.
Import with import com.alibaba.dashscope.audio.omni.OmniRealtimeCallback;.
| Method | Parameter | Return value | Description |
|---|
| Method | Parameter | Return value | Description |
java public void onOpen() | None | None | Called immediately after a connection is established with the server. |
java public abstract void onEvent(JsonObject message) | message: The server response event. | None | Contains interface call responses and model-generated text and audio. See Server events. |
java public abstract void onClose(int code, String reason) | code: The status code for closing the WebSocket. reason: The reason for closing the WebSocket. | None | Called after the connection to the server is closed. |
FAQ
Q: How are the input audio and images aligned?
The Qwen-Omni real-time model uses the audio stream as the input timeline. Images are inserted into the audio stream based on the time they are sent. You can add images at any point in the audio timeline.
In real-time interaction scenarios, you can enable or disable video input at any time.
Q: What is the recommended frequency for inputting images and audio?
In real-time interaction scenarios, we recommend that you send images at a frame rate of 1 or 2 fps and send audio in 100 ms packets.
Q: What are the differences between the two modes of the turn_detection switch?
When turn_detection is enabled, it supports two modes: server_vad and semantic_vad:
- Enable "turn_detection":
Input state: The cloud-based VAD detects the end of a sentence in the input audio and immediately triggers Qwen-Omni inference to return the reply text and speech.
Response state: In this state, you can continue to send audio and video input without interruption while the model is responding. After the response is complete, the state returns to the input state to await the next speech input.
Interruption: If the user starts speaking while the model is responding, an interruption is triggered. The service immediately stops the current response and switches to the input state.
- Disable "turn_detection":
You must manually determine the end of a round of audio and video input and trigger Qwen-Omni inference to obtain a response using commit and create_response.
While the model is responding, you must stop sending audio and video input. You can resume input for the next round only after the model has finished responding.
You must use the cancel_response method to interrupt the model's response.
Note that when turn_detection is enabled, you can still actively trigger a response using commit and create_response, and actively interrupt it using cancel_response.
Q: Why do I need to select another model for input_audio_transcription?
Qwen-Omni real-time is an end-to-end multimodal model. Its text output is a response to the input, not a direct transcription of the input audio. A separate ASR model is required for transcription. Currently, the model is determined by the built-in configuration and is not configurable.
Previous: Python SDKNext: Interaction flow
Is this page helpful?
Prerequisites
Getting started
Request parameters
Key interfaces
OmniRealtimeConversation class
Callback interface (OmniRealtimeCallback)
FAQ
Contact Us
Sales Support
Live-chat with our sales team or get in touch with a business development professional in your region.
Contact Sales
Technical Support
Open a ticket and get quick help from our technical team.
Open a Ticket >
Connect & Report Abuse
We look forward to your suggestion.
Post a Suggestion > Report Abuse >
\ \ Hi, I'm Alibaba Cloud AI Assistant!\ \ I can help with questions and solutions.
Why Alibaba Cloud
About Alibaba Cloud
Asia Accelerator
Our Global Network
Global Offices
Trust Center
Case Studies
Analyst Reports
Products & Pricings
Pricing Calculator
ECS
SAS
Model Studio
Database
Security
SMS
Solutions
Financial Services
Retail Services
Media Services
Gaming Services
ISV Solutions
Engage
Developer Community
Partner Network
Startups
Marketplace
Join Alibaba Cloud
Resources & Support
Developer Learning Hub
Documentation Center
Training & Certification
Service Notices
Submit a Ticket
Security Report
Qwen Cloud
Careers About Us Privacy Policy Legal Integrity Compliance Reporting Channel Service Notices Links
© 2009-2026 Copyright by Alibaba Cloud All rights reserved
- YouTube
- TikTok
- contact.us@alibabacloud.com
- Call Us Now
- Discord
© 2009-2026 Copyright by Alibaba Cloud All rights reserved
Careers About Us Privacy Policy Legal Integrity Compliance Reporting Channel Service Notices Links