An artificial intelligence talk at the VU Amsterdam about defining spacial representations for the meaning of sounds

1. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
By Benjamin Timmermans
In collaboration with Emiel van Miltenburg

2. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
PERCEPTIONS

3. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
CONTRADICTIONS EXIST

4. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
CONTRADICTIONS EXIST

5. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
THERE IS NO RIGHT OR WRONG

6. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
WHAT DO YOU HEAR?

7. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
“This is the sound of a piece of A4 paper being cut with scissors. The paper was wedged
between the capsules of a Rode NT-4 stereo microphone and so was in contact with
them.”
Author tags:
Scissors Stereo Cutting Foley Paper
AMBIGUOUS SOUND

8. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
PROBLEMS
Incomplete descriptions

9. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Experts describe different things then what people are
interested in
PROBLEMS

10. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
PROBLEMS:
POOR SEARCH RESULTS

11. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
WHAT IS SEARCHED FOR
• Top 5000 search queries of freesound.org
• Representing 5,6 million searches during 5 months

12. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
HYPOTHESIS:
THE CROWD CAN PROVIDE BETTER PERCEPTUAL
DESCRIPTIONS THAN EXPERTS

13. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• 2148 sounds from freesound.org
• 19.098 tags by experts
• 2.008 unique terms
• Avg 8,9 terms per sound
DATASET

14. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
ANNOTATION TASK

15. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• CrowdTruth + CrowdFlower
• US/UK/AUS/CA crowd workers
• 10 annotations for each sound
• 3 sounds in one task
• $ 0.02 reward
EXPERIMENTAL SETUP

16. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Three versions:
• 38 x 3 sounds < 30 sec.
• 300 x 3 sounds
• short: < 1 sec.
• med: 5 to 6 sec.
• long: 17 to 21 sec.
• 378 x 3 sounds < 1 sec.
DATASET

17. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• No phonetic words
• No sentences
• Correct spelling
• Comma separated tags
• Infinite number of tags
TASK INSTRUCTIONS

18. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• Remove spammers using CrowdTruth metrics
• Clean up non-alphanumeric characters
• Loud! -> loud
• Spell correction
• Barkking -> barking
• Cluster on morphology, substrings, word order
• Walk, walks, walking -> walk
• Dog, Fat dog -> Fat dog
• Running dog = Dog running
POST-PROCESSING

19. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• 573 workers
• 30.289 crowd tags
• 7.327 unique terms
• Avg 15 terms per sound
• 5% spam removed
RESULTS

20. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Experts
EXPERTS VS. CROWD
Crowd

21. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
VOCABULARY OVERLAP

22. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
TERMS ONLY USED BY CROWD

23. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
WHAT IS SEARCHED BUT NOT FOUND

24. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Expert:
Scissors Stereo Cutting Foley Paper
Crowd:
WHAT THE CROWD HEARD
Knocking (3x)
Stamping
Stomping
Hammering
Clattering
Drumming (3x)
Drum (2x)
Tapping
Drum beat
Beat
Bang (2x) Music
Irregular
No fixed rhythm
Hollow
Loud
Thud

25. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Avg terms per sound in search data:
• Crowd: 10.38
• Experts: 7.01
• Crowd + Expert: 16.63
SEARCH IMPROVEMENT

26. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
FROM MOST “CAR” TO LEAST “CAR”

27. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• Expert sound tags often do not describe what is heard
• Crowd can efficiently generate useful sound tags
• Crowd provided better perceptual descriptions
• Perceptual descriptions improve search
CONCLUSION

28. Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• Investigate the type of content described with sound tags
• Specificity
• Subjectivity
• Further investigate influence of sound length
• Improve search for sounds
FUTURE WORK