What does a genre finder’s confidence score really mean?
A genre detector says Pop at 18%. Is it unsure, or simply careful? We tested genre tagging on 999 labelled songs to find out what these percentages really predict, which genres get mixed up and how to read a result you can rely on.
Two very different percentages
Many genre detectors work as taggers. They score every genre they know separately, so a song can earn 40% for rock and 30% for blues at the same time. These scores do not add up to 100%, and they are not the chance that the answer is right. They describe how strongly the music resembles examples of each genre.
In our test, the top tag of a typical song scored only 27%, and that raw score said little about whether the genre was right. Songs whose top tag scored under 15% were right 63% of the time; songs scoring 25–35% were right 60% of the time. Only above 50% did accuracy clearly rise, to 79%.
A confidence you can act on answers a different question: when a detector gives this reading, how often is it right? That can only be measured on songs whose genre is already known.
A clear lead is what predicts a right answer
How clearly the top genre leads mattered far more than its raw score. We divided the top genre’s score by the total of all genre scores. When the top genre held less than 40% of that total, it was right 47% of the time. When it held 80% or more, it was right 99% of the time.
| Top genre’s share of all genre scores | Songs | Top genre right |
|---|---|---|
| Under 40% | 462 | 47% |
| 40–50% | 192 | 63% |
| 50–60% | 104 | 80% |
| 60–70% | 58 | 90% |
| 70–80% | 45 | 93% |
| 80% or more | 138 | 99% |
The confidence in Music Genre Finder reports this kind of measured rate: how often the top genre was right in testing at the same level of lead. A reading of 55% means the top genre was right a little more than half the time at that level. A reading of 95% means it was almost always right.
What we tested
We used GTZAN, a public research collection of 999 thirty-second song clips in ten genres: blues, classical, country, disco, hip hop, jazz, metal, pop, reggae and rock. The detection chose freely among 49 genres. We counted an answer as right when it named the labelled genre, one of its sub-genres or a parent style. Heavy metal counts for metal and swing for jazz, but funk does not count for disco.
| Measure | Result |
|---|---|
| Top genre right | 65.4% |
| Right genre among the top two | 79.0% |
| Right genre among the top three | 85.9% |
Most misses are neighbouring genres
| Labelled genre | Top genre right | Most common other answer |
|---|---|---|
| Classical | 99% | Almost never wrong |
| Jazz | 96% | Classical (3 songs) |
| Metal | 80% | Rock and roll (14 songs) |
| Hip hop | 72% | Funk (15 songs) |
| Country | 71% | Christian or soul (6 songs each) |
| Rock | 66% | Soul (10 songs) |
| Reggae | 64% | Soul or funk (7 songs each) |
| Blues | 37% | Swing (13 songs) |
| Pop | 37% | R&B (28 songs) |
| Disco | 32% | Funk (40 songs) |
Genre names are history as much as sound. Disco grew out of funk and soul, so 40 disco songs were read as funk and 14 as soul. Pop overlapped with R&B, and blues with swing, psychedelic rock and country. These answers are wrong for a label, but they are often reasonable to a listener.
Classical and jazz were almost always right, because their instruments and textures are distinctive. The genres that were hardest to name were the ones that sit between neighbours, which is why the second and third answers deserve as much attention as the first.
Genres that detectors know less well
Detectors learn from labelled examples, and their genre lists reflect the music those examples came from. A style without a label of its own is matched to the nearest familiar pattern. In our checks beyond the test collection, Despacito, a reggaeton hit, was read as afrobeat by most of the taggers we tried, and an Indian pop song was read as South Indian classical music.
For regional styles and newer scenes, use the result to find neighbouring genres rather than as the final name. Your own knowledge of the music, the artist and the audience is the better guide.
Mood is harder still. Mood tags were weak for most songs: only 8% of the test songs had any mood tag above 10%. Treat a mood label as a hint about character, not a measurement.
How to read a genre result
- Look at the top three genres, not only the first. The right genre was among the top three for 86% of the test songs.
- Check whether they agree. Rock, heavy metal and grunge together point firmly to one family; rock, reggae and jazz together mean the detector is unsure.
- Read the confidence as a rate. Below about 60%, treat the top genre as one candidate among several. Above 90%, it was almost always right in testing.
- For release metadata, pick a primary and a secondary genre from the top answers that match what you hear. Distributors usually ask for both.
- For playlists and libraries, combine genre with tempo and key. Two songs from neighbouring genres often sit well together when their tempo and key are compatible.
Music Genre Finder shows the top genre with its confidence, the next genres with their share of the match, and the song’s tempo, key and energy.
For tempo and key, read why a BPM finder shows half or double the tempo and how accurate automatic song key detection is.
References
GTZAN: Tzanetakis and Cook, “Musical genre classification of audio signals”, IEEE Transactions on Speech and Audio Processing, 2002.
Known limits of GTZAN: Sturm, “The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use”, 2013.