AnySynth: Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance

Anonymous Authors

1. Model Overview

AnySynth is a novel neural synthesizer for zero-shot instrument cloning. Given a target MIDI sequence and a short reference audio–MIDI pair, AnySynth renders the target sequence using the exact acoustic identity of an unseen reference instrument — without any fine-tuning or instrument-specific training.

Unlike prior approaches that compress reference audio into a fixed-length timbre embedding (losing fine-grained details like transients, vibrato, and room acoustics), AnySynth reformulates the task as an in-context learning problem. The uncompressed reference mel spectrogram is fed directly as a prefix to a Diffusion Transformer (DiT) backbone, whose self-attention mechanism dynamically retrieves acoustic details without information loss.

During Inference, we use Asymmetric Hierarchical CFG to guide the reverse ODE process.

AnySynth Architecture

Figure 1: Overview of the AnySynth architecture.

2. Audio Demonstrations

Below we compare AnySynth against three baselines — AnySynth (CLAP) (our model with a CLAP embedding instead of in-context learning), CTD (Control-Transfer-Diffusion), and TokenSynth — across four datasets. Each model receives the same reference prompt audio (at 3s, 8s, or 15s) and generates a 5-second audio clip for the same target MIDI. The ground truth is provided as reference.