[me]

Brian McFee brian.mcfee@nyu.edu

ORCID iD iconhttps://orcid.org/0000-0001-6261-9747

Assistant Professor of Music Technology and Data Science at New York University

Music and Performing Arts Professions/Music and Audio Research Lab (MARL) and Center for Data Science

I develop machine learning tools to analyze music and multimedia data. This includes recommender systems, image and audio analysis, similarity learning, cross-modal feature integration, and automatic annotation.

For a full history, here's my curriculum vitæ.

Ph.D. Students

Books

Publications

2024
bib | pdf
Spatial Scaper: A Library to Simulate and Augment Soundscapes for Sound Event Localization and Detection in Realistic Rooms
International conference on acoustics, speech and signal processing (ICASSP).
2024
bib | pdf
Self-Supervised Learning of Multi-level Audio Representations for Music Segmentation
Buisson, M., McFee, B., Essid, S., and Crayencour, H.
IEEE Transactions on Audio, Speech and Language Processing (to appear)
2023
bib | pdf
A Repetition-based Triplet Mining Approach for Music Segmentation
Buisson, M., McFee, B., Essid, S., and Crayencour, H.
International society for music information retrieval (ISMIR) conference.
2023
code | bib | pdf
Transfer Learning and Bias Correction with Pre-trained Audio Embeddings
Wang, C.Richard, G., and McFee, B.
International society for music information retrieval (ISMIR) conference.
2023
bib | pdf
Leveraging Geometrical Acoustic Simulations of Spatial Room Impulse Responses for Improved Sound Event Detection and Localization
Ick, C. and McFee, B.
Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)
2023
bib | pdf
Efficient Evaluation Algorithms for Sound Event Detection
Lostanlen, V. and McFee, B.
Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)
2023
bib | pdf
Foley Sound Synthesis at the DCASE 2023 Challenge
Choi, K.Im, J.Heller, L.M., McFee, B., Imoto, K., Okamoto, Y., Lagrange, M., and Takamichi, S.
Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)
2023
code | bib | pdf
Automatic recognition of cascaded guitar effects
Guo, J. and McFee, B.
International conference on digital audio effects (DAFx)
2022
bib | pdf
Learning multi-level representations for hierarchical music structure analysis
Buisson, M., McFee, B., Essid, S., and Crayencour, H.
International society for music information retrieval (ISMIR) conference.
2021
web | bib |
Automatic Hierarchy Expansion for Improved Structure and Chord Evaluation
Kinnaird, K. and McFee, B.
Transactions of the International Society for Music Information Retrieval
2021
web | bib |
Sound Event Detection in Urban Audio With Single and Multi-Rate PCEN
Ick, C. and McFee, B.
International conference on acoustics, speech and signal processing (ICASSP).
2021
web | bib |
Multi-Task Self-Supervised Pre-Training for Music Classification
Wu, H.H., Kao, C.C., Tang, Q., Sun, M., McFee, B., Bello, J.P., and Wang, C.
International conference on acoustics, speech and signal processing (ICASSP).
2021
web | bib |
Interactive Learning of Signal Processing Through Music: Making Fourier Analysis Concrete for Students
Müller, M., McFee, B., and Kinnaird, K.
IEEE Signal Processing Magazine
2020
web | bib | pdf
Audio-Based Music Structure Analysis: Current Trends, Open Challenges, and Applications
Transactions of the International Society for Music Information Retrieval
2020
bib | pdf
Multiple F0 estimation in vocal ensembles using convolutional neural networks
Cuesta, H., McFee, B., and Gómez, E.
International society for music information retrieval (ISMIR) conference.
2020
bib | pdf
Entrofy your cohort: a transparent method for diverse cohort selection
Huppenkothen, D., McFee, B., and Norén, L.
PLOS ONE
2020
bib | pdf
Learning the helix topology of musical pitch
International conference on acoustics, speech and signal processing (ICASSP).
2019
bib | pdf
Improving structure evaluation through automatic hierarchy expansion
McFee, B. and Kinnaird, K.
International society for music information retrieval (ISMIR) conference.
2019
bib | pdf
Voice anonymization in urban sound recordings
International workshop on machine learning for signal processing (MLSP).
2019
bib | pdf
Enhanced hierarchical music structure annotations via feature level similarity fusion
Tralie, C.J. and McFee, B.
International conference on acoustics, speech and signal processing (ICASSP).
2019
bib | pdf
A music structure informed downbeat tracking system using skip-chain conditional random fields and deep learning
Fuentes, M., McFee, B., Crayencour, H., Essid, S., and Bello, J.P.
International conference on acoustics, speech and signal processing (ICASSP).
2019
bib | pdf
Open source practices for music signal processing research
IEEE Signal Processing Magazine
2019
web | bib | pdf
Per-channel energy normalization: why and how
IEEE Signal Processing Letters
2018
code | bib | pdf
Adaptive pooling operators for weakly labeled sound event detection
McFee, B., Salamon, J., and Bello, J.P.
IEEE Transactions on Audio, Speech and Language Processing
2018
code | data | bib | pdf
OpenMIC-2018: An open dataset for multiple instrument recognition
Humphrey, E.Durand, S., and McFee, B.
19th International Society for Music Information Retrieval (ISMIR) conference
2018
bib | pdf
Analysis of common design choices in deep learning systems for downbeat tracking
Fuentes, M., McFee, B., Crayencour, H., Essid, S., and Bello, J.P.
19th International Society for Music Information Retrieval (ISMIR) conference
2018
bib | pdf
Bubble cooperative networks for identifying important speech cues
Trinh, V.A., McFee, B., and Mandel, M.
InterSpeech
2017
code | web | bib | pdf
Evaluating hierarchical structure in music annotations
McFee, B., Nieto, O.Farbood, M., and Bello, J.P.
Frontiers in Psychology
2017
slides | code | bib | pdf
Structured training for large-vocabulary chord recognition
McFee, B. and Bello, J.P.
18th International Society for Music Information Retrieval (ISMIR) conference
2017
code | bib | pdf
Best student paper award
Deep salience representations for F0 estimation in polyphonic music
18th International Society for Music Information Retrieval (ISMIR) conference
2017
bib | pdf
Statistical methods for scene and event classification
McFee, B.
Computational Analysis of Sound Scenes and Events
2016
code | bib | pdf
resampy: efficient sample rate conversion in python
McFee, B.
The Journal of Open Source Software
2016
slides | bib | pdf
A plan for sustainable MIR evaluation
McFee, B., Humphrey, E., and Urbano, J.
17th International Society for Music Information Retrieval (ISMIR) conference
2015
slides | code | bib | pdf
A software framework for musical data augmentation
McFee, B., Humphrey, E., and Bello, J.P.
16th International Society for Music Information Retrieval (ISMIR) conference
2015
poster | bib | pdf
Hierarchical evaluation of segment boundary detection
McFee, B., Nieto, O., and Bello, J.P.
16th International Society for Music Information Retrieval (ISMIR) conference
2015
slides | code | bib | pdf
librosa: Audio and Music Signal Analysis in Python
14th annual Scientific Computing with Python conference (SciPy)
2014
video | slides | code | bib | pdf
Best oral presentation award
Analyzing song structure with spectral clustering
McFee, B. and Ellis, D.P.W.
15th International Society for Music Information Retrieval (ISMIR) conference
2014
code | bib | pdf
Best poster presentation award
mir_eval: a transparent implementation of common MIR metrics
15th International Society for Music Information Retrieval (ISMIR) conference
2014
bib | pdf
Codebook-based audio feature representation for music information retrieval
Vaizman, Y., McFee, B., and Lanckriet, G.R.G.
IEEE Transactions on Audio, Speech and Language Processing
2014
bib | pdf
Speech enhancement by low-rank and convolutive dictionary spectrogram decomposition
Chen, Z., McFee, B., and Ellis, D.P.W.
Interspeech
2014
code | bib | pdf
Learning to segment songs with ordinal linear discriminant analysis
McFee, B. and Ellis, D.P.W.
International conference on acoustics, speech and signal processing (ICASSP)
2014
code | bib | pdf
Better beat tracking through robust onset aggregation
McFee, B. and Ellis, D.P.W.
International conference on acoustics, speech and signal processing (ICASSP)
2013
bib | pdf
Iterative category discovery via multiple kernel metric learning
International Journal of Computer Vision
2013
code | bib | pdf
Robust structural metric learning
Lim, D.K.H., McFee, B., and Lanckriet, G.R.G.
30th International Conference on Machine Learning (ICML)
2012
slides | code | data | bib | pdf
Hypergraph models of playlist dialects
McFee, B. and Lanckriet, G.R.G.
13th International Society for Music Information Retrieval (ISMIR) conference
2012
data | bib | pdf
How significant is statistically significant? The case of audio music similarity and retrieval
Urbano, J.Downie, J.S., McFee, B., and Schedl, M.
13th International Society for Music Information Retrieval (ISMIR) conference
2012
data | bib | pdf
The Million Song Dataset Challenge
4th International Workshop on Advances in Music Information Research (AdMIRe)
2012
bib | pdf
Learning content similarity for music recommendation
IEEE Transactions on Audio, Speech and Language Processing
2011
poster | data | bib | pdf
Best poster presentation award
The natural language of playlists
McFee, B. and Lanckriet, G.R.G.
12th International Society for Music Information Retrieval (ISMIR) conference
2011
code | bib | pdf
Large-scale music similarity search with spatial trees
McFee, B. and Lanckriet, G.R.G.
12th International Society for Music Information Retrieval (ISMIR) conference
June, 2011
bib | pdf
From region similarity to category discovery
IEEE conference on Computer Vision and Pattern Recognition (CVPR)
February, 2011
bib | pdf
Learning multi-modal similarity
McFee, B. and Lanckriet, G.R.G.
Journal of Machine Learning Research (JMLR)
February, 2011
bib | pdf
Contextual object localization with multiple kernel nearest neighbor
IEEE Transactions on Image Processing (TIP)
2010
bib | pdf
Learning similarity from collaborative filters
11th International Society for Music Information Retrieval (ISMIR) conference
2010
bib | pdf
Collaborative filtering based on P2P networks
11th International Society for Music Information Retrieval (ISMIR) conference
2010
poster | code | data | bib | pdf
Metric learning to rank
McFee, B. and Lanckriet, G.R.G.
Twenty-seventh International Conference on Machine Learning (ICML)
2010
bib | pdf
Multi-class object localization by combining local contextual interactions
IEEE conference on Computer Vision and Pattern Recognition (CVPR)
2009
poster | data | bib | pdf
Best presentation award
Heterogeneous embedding for subjective artist similarity
McFee, B. and Lanckriet, G.R.G.
10th International Society for Music Information Retrieval (ISMIR) conference
2009
Partial order embedding with multiple kernels
McFee, B. and Lanckriet, G.R.G.
Twenty-sixth International Conference on Machine Learning (ICML)

Software

autopool
automatic pooling for multiple instance learning
crema
convolutional and recurrent estimators for music analysis
pumpp
practically universal music pre-processor
pescador
stream sampling for iterative learning algorithms
amen
algorithmic music remixing.
resampy
efficient audio resampling in Python.
muda
Musical Data Augmentation.
JAMS
a JSON Annotated Music Specification. v0.2 technical report
Ordinal LDA
Python (sklearn) implementation of ordinal linear discriminant analysis.
librosa
A python package for music and audio signal analysis.
MLR
MATLAB implementation of metric learning to rank.
Hypergraph playlists
Python implementation of the model from this paper.
Spatial trees
Python implementation of spatial trees for approximate nearest neighbor search, as used in this paper.

More projects can be found on my GitHub profile.

Data

Open-MIC 2018
Audio and partial instrument annotations for 20,000 10-second clips
MSD Challenge
Large-scale music recommendation on the Million Song Dataset. See also the year 1 test set.
AotM-2011
Annotated playlists from Art of the Mix, indexed to the Million Song Dataset.
AotM-2003
An earlier collection of playlists from Art of the Mix, also indexed to the Million Song Dataset.
aset400 kernels
Kernel matrices for aset400 artist similarity experiments
eHarmony
Matchings and anonymized features for several hundred thousand eHarmony users.

Teaching

Fall 2020
(NYU) DS-GA 3001: Search and Discovery
Spring 2019, 2020, 2021
(NYU) DS-GA 1004: Big Data
Fall 2018, 2019, 2020
(NYU) MPATE-GE 2599: Fundamentals of Digital Signals Theory I
Spring 2014
(Columbia) EECS E6891: Reproducing computational results

Hack-day projects

The Keplerphone
Music synthesis from the Kepler satellite. Shredding exoplanet rock.
Yankomatic
Automatically generate song title parodies using rhyming food words.
Handmadedness
Calculate how hand-made a song sounds by measuring repetition patterns and tempo stability.
Frankenmasher 2000
Isolate and recombine vocals and instruments from different songs to create mash-ups of hypothetical super-groups.
Mend-a-break
Resynthesize any audio clip using sparse convolutions of the Amen Break.
Auto-chip-tune
Generate Nintendo chip-tune programs from audio analysis.

Version 2

See my hack-day blog for more projects.