SoundML
Using built-in machine learning to classify sounds.
A SoundML Analyzer object observes audio buffers, and maintains an internal ML analysis, reporting onUpdate for any SoundGroup of interest. A SoundGroup represents one or more Sound objects, any of which matching above their threshold consitutes a match. These Sound objects represent a specific ML recognition.
The SoundType enum represents all known labels of the SNClassifierIdentifier.version1 model as of iOS 15.0.
Usage
Setup
First, create an Analyzer object, and, optionally, adjust its properties:
import SoundML
let analyzer = Analyzer()
analyzer.windowDuration = 0.75
Next, set one or more SoundGroup instances you'd like to observe.
Let's imagine you're working on an app that keeps track of when you're snoring, and when you're talking in your sleep at night:
analyzer.soundGroups = [
SoundGroup(.speech, threshold: 0.7),
SoundGroup(.snoring, threshold: 0.4)
]
You can also include more than one sound in each group, with different threshold. Imagine you wanted to count either speech or whispering:
analyzer.soundGroups = [
SoundGroup(.snoring, threshold: 0.4),
SoundGroup(id: "talking", sounds: [
Sound(.speech, threshold: 0.7),
Sound(.whispering, threshold: 0.5)
])
]
In this case, either speech or whispering, at confidence levels of 70% and 50%, respectively, would trigger the group with id talking.
Processing
As you receive audio buffers, from an AVAudioEngine, for example, pass them to the analyzer:
audioEngine.inputNode.installTap(onBus: 0, bufferSize: 2048, format: nil) { [weak self] buffer, time in
self?.analyzer.process(buffer: buffer, time: time)
}
Analysis
Asynchronously, as the internal SNAudioStreamAnalyzer builds up enough samples to process, the Analyzer will call its onUpdate handler.
This contains a Bool representing whether a match occurred, as well as an array of matches for any sounds that successfully matched:
analyzer.onUpdate = { [weak self] _, groups in
guard let groups = groups else {
print("No matches for this frame.")
return
}
for match in groups {
print("Matched: \(match.sound.label)\n\(Int(match.confidence * 100))%)
}
}
These match objects contain the SoundGroup that matched the recently processed sound, the Sound within that group that matched most highly, and the confidence of the match. This value must be above that sound's threshold.
Note that multiple matches can occur at the same time. If, for example, two people were sleeping, with one talking, and the other snoring, as often happens.
Known Sounds
Below is an alphabetized list of sounds the Analyzer can recognize. You may pass these as strings, or use the SoundType enum.
accordionacoustic_guitarair_conditionerair_hornaircraftairplanealarm_clockambulance_sirenapplauseartillery_firebabblebaby_cryingbaby_laughterbagpipesbanjobasketball_bouncebass_drumbass_guitarbassoonbathtub_filling_washingbattle_crybee_buzzbeepbellbelly_laughbicyclebicycle_bellbirdbird_chirp_tweetbird_flappingbird_squawkbird_vocalizationbitingblenderboat_water_vehicleboilingbooingboombowed_string_instrumentbowling_impactbrass_instrumentbreathingburpbuscameracar_horncar_passing_bycatcat_meowcat_purrcellochainsawchattercheeringchewingchickenchicken_cluckchildren_shoutingchimechoir_singingchopping_foodchopping_woodchuckle_chortlechurch_bellcivil_defense_sirenclappingclarinetclickclockcoin_droppingcoughcow_moocowbellcoyote_howlcricket_chirpcrow_cawcrowdcrumpling_crinklingcrushingcrying_sobbingcutlery_silverwarecymbaldidgeridoodisc_scratchingdishes_pots_pansdogdog_barkdog_bow_wowdog_growldog_howldog_whimperdoordoor_belldoor_slamdoor_slidingdouble_bassdrawer_open_closedrilldrumdrum_kitduck_quackelectric_guitarelectric_pianoelectric_shaverelectronic_organelk_bugleemergency_vehicleengineengine_accelerating_revvingengine_idlingengine_knockingengine_startingeruptionfinger_snappingfirefire_cracklefire_engine_sirenfirecrackerfireworksflutefly_buzzfoghornfowlfrench_hornfrogfrog_croakfrying_foodgarglinggaspgigglingglass_breakingglass_clinkglockenspielgonggoose_honkguitarguitar_strumguitar_tappinggunshot_gunfiregurglinghair_dryerhammerhammond_organharmonicaharpharpsichordhedge_trimmerhelicopterhi_hathiccuphorse_clip_clophorse_neighhumminginsectkeyboard_musicalkeys_janglingknocklaughterlawn_mowerlion_roarliquid_drippingliquid_filling_containerliquid_pouringliquid_sloshingliquid_splashingliquid_sprayingliquid_squishingliquid_trickle_dribblemallet_percussionmandolinmarimba_xylophonemechanical_fanmicrowave_ovenmosquito_buzzmotorboat_speedboatmotorcyclemusicnose_blowingoboeoceanorchestraorganowl_hootpercussionperson_runningperson_shufflingperson_walkingpianopig_oinkpigeon_dove_cooplaying_badmintonplaying_hockeyplaying_squashplaying_table_tennisplaying_tennisplaying_volleyballplucked_string_instrumentpolice_sirenpower_toolpower_windowsprinterrace_carrail_transportrailroad_carrainraindroprappingratchet_and_pawlrattle_instrumentreverse_beepsringtonerooster_crowrope_skippingrowboat_canoe_kayaksailingsawsaxophonescissorsscreamingscuba_divingsea_wavessewing_machinesheep_bleatshofarshoutsighsilencesingingsinging_bowlsink_filling_washingsirensitarskateboardskiingslap_smackslurpsmoke_detectorsnake_hisssnake_rattlesnare_drumsneezesnickersnoringspeechsqueaksteel_guitar_slide_guitarsteelpanstream_burblingsubway_metrosynthesizertablatambourinetaptearingtelephonetelephone_bell_ringingthereminthump_thudthunderthunderstormticktick_tocktimpanitoilet_flushtoothbrushtraffic_noisetraintrain_horntrain_wheels_squealingtrain_whistletrombonetrucktrumpettuning_forkturkey_gobbletypewritertypingtyping_computer_keyboardukuleleunderwater_bubblingvacuum_cleanervehicle_skiddingvibraphoneviolin_fiddlewaterwater_pumpwater_tap_faucetwaterfallwhale_vocalizationwhisperingwhistlingwhoosh_swoosh_swishwindwind_chimewind_instrumentwind_noise_microphonewind_rustling_leaveswood_crackingwritingyellyodelingzipperzither
Further Reading
This project was adapted from Apple’s excellent sample project.
GitHub
| link |
| Stars: 4 |
| Last commit: 3 weeks ago |
Something's broken? Yell at me @ptrpavlik. Praise and feedback (and money) is also welcome.
Release Notes
Swiftpack is being maintained by Petr Pavlik | @ptrpavlik | @swiftpackco | API | Analytics