You are currently logged in as an
Institutional Subscriber.
If you would like to logout,
please click on the button below.
Home / Publications / E-library page
Only AES members and Institutional Journal Subscribers can download
This work presents a framework to impose the audio effects and production style from one recording to another by example with the goal of simplifying the audio production process. A deep neural network was trained to analyze an input recording and a style reference recording and predict the control parameters of audio effects used to render the output. In contrast to past work, this approach integrates audio effects as differentiable operators, enabling backpropagation through audio effects and end-to-end optimization with an audio-domain loss. Pairing this framework with a self-supervised training strategy enables automatic control of audio effects without the use of any labeled or paired training data. A survey of existing and new approaches for differentiable signal processing is presented, demonstrating how each can be integrated into the proposed framework along with a discussion of their trade-offs. The proposed approach is evaluated on both speech and music tasks, demonstrating generalization both to unseen recordings and even sample rates different than those during training. Convincing production style transfer results are demonstrated with the ability to transform input recordings to produced recordings, yielding audio effect control parameters that enable interpretability and user interaction.
Author (s): Steinmetz, Christian J.; Bryan, Nicholas J.; Reiss, Joshua D.
Affiliation:
Centre for Digital Music, Queen Mary University of London, London, UK; Adobe Research, San Fransico, CA; Centre for Digital Music, Queen Mary University of London, London, UK
(See document for exact affiliation information.)
Publication Date:
2022-09-06
Import into BibTeX
Permalink: https://aes2.org/publications/elibrary-page/?id=21883
(450KB)
Click to purchase paper as a non-member or login as an AES member. If your company or school subscribes to the E-Library then switch to the institutional version. If you are not an AES member Join the AES. If you need to check your member status, login to the Member Portal.
Steinmetz, Christian J.; Bryan, Nicholas J.; Reiss, Joshua D.; 2022; Style Transfer of Audio Effects with Differentiable Signal Processing [PDF]; Centre for Digital Music, Queen Mary University of London, London, UK; Adobe Research, San Fransico, CA; Centre for Digital Music, Queen Mary University of London, London, UK; Paper ; Available from: https://aes2.org/publications/elibrary-page/?id=21883
Steinmetz, Christian J.; Bryan, Nicholas J.; Reiss, Joshua D.; Style Transfer of Audio Effects with Differentiable Signal Processing [PDF]; Centre for Digital Music, Queen Mary University of London, London, UK; Adobe Research, San Fransico, CA; Centre for Digital Music, Queen Mary University of London, London, UK; Paper ; 2022 Available: https://aes2.org/publications/elibrary-page/?id=21883
@article{steinmetz2022style,
author={steinmetz christian j. and bryan nicholas j. and reiss joshua d.},
journal={journal of the audio engineering society},
title={style transfer of audio effects with differentiable signal processing},
year={2022},
volume={70},
issue={9},
pages={708-721},
month={september},}