## Content

- Summary of the compared models
- Example results on real-world audio mixtures
  1. “Someone is coughing while a woman speaks, and a cat meows later.”
  2. “A woman talks in a microphone, while several children yelling in the background.”
  3. “A woman speaks while music is being played, with sounds of frying foods in the background."
  4. “A beep sound followed by wind blows.”
  5. “Someone is playing guitar with whistle blowing, and a man talks afterwards.”
  6. “Phone rings followed by a woman talks."

## Summary of the compared models

- OpenSep (Proposed): This is our proposed model. We use the end-to-end model without manual prompting or text conditioning. The best match between parsed sources and manually curated condition prompts are used for demonstration.
- AudioSep: AudioSep1 is a text-conditioned audio separator used as a baseline. We directly provide manually curated source prompts for conditional separation.
- CLIPSep: CLIPSep2 is a text-conditioned audio separator used as a baseline. We directly provide manually curated source prompts for conditional separation.
- MixIT+PIT: This is an unconditional baseline combining both mixture invariant training, MixIT3 and permutation invariant training, PIT4. We manually align the best matched predictions with parsed sources, since these models rely on post-processing to parse source entities from predictions.

## Example results on real-world audio mixtures

### Example 1 – "Someone is coughing while a woman speaks, and a cat meows later.”

- Source1: “A woman speaks”
- Source 2: “A cat meows”
- Source 3: “Someone is coughing”

Mixture
Your browser does not support the audio element.

OpenSep (Source 1) | OpenSep (Source 2) | OpenSep (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

AudioSep (Source 1) | AudioSep (Source 2) | AudioSep (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

CLIPSep (Source 1) | CLIPSep (Source 2) | CLIPSep (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

MixIT+PIT (Source 1) | MixIT+PIT (Source 2) | MixIT+PIT (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

> * All baselines show large spectral overlap of “woman talking” sound in other two source predictions. OpenSep precisely disentangles all three sources minimizing the spectral overlap across sources, while preserving spectral details.

### Example 2 – "A woman talks in a microphone, while several children yelling in the background."

- Source1: “A woman talks in a microphone”
- Source 2: “Several children yelling”

Mixture
Your browser does not support the audio element.

OpenSep (Source 1) | OpenSep (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

AudioSep (Source 1) | AudioSep (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

CLIPSep (Source 1) | CLIPSep (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

MixIT+PIT (Source 1) | MixIT+PIT (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

> * For the dominant noisy sound of "children yelling", all baselines can hardly separate the "woman talking" sound. OpenSep significantly reduces noise in "woman talking", while preserving spectral details of noisy "children yelling" sound.

### Example 3 – "A woman speaks while music is being played, with sounds of frying foods in the background."

- Source1: “A woman speaks”
- Source 2: “Music is being played”
- Source 3: “Frying foods”

Mixture
Your browser does not support the audio element.

OpenSep (Source 1) | OpenSep (Source 2) | OpenSep (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

AudioSep (Source 1) | AudioSep (Source 2) | AudioSep (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

CLIPSep (Source 1) | CLIPSep (Source 2) | CLIPSep (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

MixIT+PIT (Source 1) | MixIT+PIT (Source 2) | MixIT+PIT (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

> * We can observe the dominant “woman talks” spectral content in “frying foods” for most baselines. However, in CLIPSep, such overlap is largely reduced, but horizontal spectral contents from “music plays” is visible. In contrast, OpenSep largely reduces such spectral overlap in all three components while preserving all details.

### Example 4 – "A beep sound followed by wind blows."

- Source1: “A beep sound”
- Source 2: “Wind blows”

Mixture
Your browser does not support the audio element.

OpenSep (Source 1) | OpenSep (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

AudioSep (Source 1) | AudioSep (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

CLIPSep (Source 1) | CLIPSep (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

MixIT+PIT (Source 1) | MixIT+PIT (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

> * In this mixture, the "beep sound" is only present at the beginning, with large noisy sound of "wind blows" over the spectrogram. Most baseline methods contain noisy spectral contents in the "beep sound", while losing spectral contents in the "wind blows" prediction. In contrast, OpenSep disentangles this noisy mixture with significant reduction of spectral overlaps.

### Example 5 – “Someone is playing guitar with whistle blowing, and a man talks afterwards.”

- Source1: “Someone is playing guitar”
- Source 2: “Whistle blowing”
- Source 3: “A man talks”

Mixture
Your browser does not support the audio element.

OpenSep (Source 1) | OpenSep (Source 2) | OpenSep (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

AudioSep (Source 1) | AudioSep (Source 2) | AudioSep (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

CLIPSep (Source 1) | CLIPSep (Source 2) | CLIPSep (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

MixIT+PIT (Source 1) | MixIT+PIT (Source 2) | MixIT+PIT (Source 3)
--- | --- | ---
Your browser does not support the audio element. | Your browser does not support the audio element. | Your browser does not support the audio element.

> * In the initial phase, the “whistle blows” and “guitar plays” sounds are present, while the “man talks” sound appears at the end. In all baselines, we can see significant spectral overlaps of the “whistle blows” and “guitar plays” sounds. In contrast, OpenSep precisely separates both of these challenging components, while also reducing background contents in the “man talks” prediction.

### Example 6 – "Phone rings followed by a woman talks."

- Source1: “Phone rings”
- Source 2: “A woman talks”

Mixture
Your browser does not support the audio element.

OpenSep (Source 1) | OpenSep (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

AudioSep (Source 1) | AudioSep (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

CLIPSep (Source 1) | CLIPSep (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

MixIT+PIT (Source 1) | MixIT+PIT (Source 2)
--- | ---
Your browser does not support the audio element. | Your browser does not support the audio element.

> * The "phone rings" sound mostly appears at the beginning followed by the "woman talks" sound at the end. Compared to all baselines, OpenSep more sharply disentangles both of these sources from this challenging mixture highlighting its effectiveness in practice.

1. Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D Plumbley, and Wenwu Wang. 2023. Separate anything you describe. arXiv preprint arXiv:2308.05037 ↩

2. DDong, H. W., Takahashi, N., Mitsufuji, Y., McAuley, J., & Berg-Kirkpatrick, T. (2022, September). CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos. In The Eleventh International Conference on Learning Representations. ↩

3. Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron Weiss, Kevin Wilson, and John Hershey. 2020. Un-supervised sound separation using mixture invariant training. Advances in Neural Information Processing, Systems, 33:3846–3857 ↩

4. Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen. 2017. Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 241–245. IEEE. ↩