Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

detect audio file tracks #1439

Merged
merged 7 commits into from
Jan 27, 2025
Merged

detect audio file tracks #1439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f0ff149
add methods for file detection
peetucket Dec 9, 2024
b807d34
call out to technical metadata service to see if file has an audio track
peetucket Dec 10, 2024
0942540
test for has_audio_track?
peetucket Dec 10, 2024
109d394
update code comment
peetucket Dec 12, 2024
778e53b
use tech metadata to check volume levels
peetucket Dec 13, 2024
e0c11da
raise an exception for missing expected audio technical metadata
peetucket Dec 16, 2024
27db579
add extra test
peetucket Dec 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 44 additions & 1 deletion lib/dor/text_extraction/speech_to_text.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,53 @@ def cleanup_s3_folder
# return a list of filenames that should be stt'd
# iterate over all files in cocina_object.structural.contains, looking at mimetypes
# return a list of filenames that are correct mimetype
# then filter out any files that either (1) do not have an audio track or (2) have audio that is mostly silent
def filenames_to_stt
stt_files.map(&:filename)
available_files = stt_files.map(&:filename)
available_files.select { |filename| has_useful_audio_track?(filename) }
end
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is what filters the files to send to the speech to text to only those with an audio track that is not mostly silent


# first verify that the file has an audio track, then check the audio metadata to determine if the audio is mostly silent
# using technical metadata generated in https://github.com/sul-dlss/technical-metadata-service/pull/572
# check the audio max_volume and mean_volume fields to determine if the audio is mostly silent
# if will raise an error if this metadata is missing
# rubocop:disable Metrics/CyclomaticComplexity
# rubocop:disable Metrics/PerceivedComplexity
# rubocop:disable Metrics/AbcSize
def has_useful_audio_track?(filename)
return false unless file_level_tech_metadata(filename)&.dig('av_metadata', 'audio_count')&.positive?

audio_metadata = file_level_tech_metadata(filename)&.dig('dro_file_parts')&.find { |parts| parts['part_type'] == 'audio' }&.dig('audio_metadata')

raise "No audio metadata found for #{filename}" unless audio_metadata
raise "Audio metadata missing max_volume and mean_volume for #{filename}" unless audio_metadata['max_volume'] && audio_metadata['mean_volume']

audio_metadata['mean_volume'] > -40 && audio_metadata['max_volume'] > -30
end
# rubocop:enable Metrics/CyclomaticComplexity
# rubocop:enable Metrics/PerceivedComplexity
# rubocop:enable Metrics/AbcSize

# return the technical metadata for a given filename
def file_level_tech_metadata(filename)
tech_metadata.find { |file| file['filename'] == filename }
end

# return the technical metadata for the object from the technical-metadata-service and parse it as json
# rubocop:disable Metrics/AbcSize
def tech_metadata
@tech_metadata ||= begin
resp = Faraday.get("#{Settings.tech_md_service.url}/v1/technical-metadata/druid/#{cocina_object.externalIdentifier}") do |req|
req.headers['Content-Type'] = 'application/json'
req.headers['Authorization'] = "Bearer #{Settings.tech_md_service.token}"
end
raise "Technical-metadata-service returned #{resp.status} when requesting techmd for #{bare_druid}: #{resp.body}" unless resp.success?

JSON.parse(resp.body)
end
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the code that does that detection by examining technical metadata

note that volume detection level technical metadata is dependent on sul-dlss/technical-metadata-service#572 being merged first and technical metadata updated on any object that needs it

end
# rubocop:enable Metrics/AbcSize

# return the s3 location for a given filename
def s3_location(filename)
File.join(job_id, filename)
Expand Down
3 changes: 0 additions & 3 deletions lib/robots/dor_repo/speech_to_text/start_stt.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,6 @@ def initialize
end

def perform_work
# TODO: Note that the `possible?` method is not complete until we further refine the mimetypes available
# see https://github.com/sul-dlss/common-accessioning/issues/1346
# and lib/dor/text_extraction/speech_to_text.rb#allowed_mimetypes
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no longer true, remove this TODO

if Dor::TextExtraction::SpeechToText.new(cocina_object:).possible?
return if object_client.version.status.open?

Expand Down
146 changes: 146 additions & 0 deletions spec/fixtures/technical_metadata/bc123df4567.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
[
{
"druid": "druid:bc123df4567",
"filename": "file1.m4a",
"filetype": "fmt/199",
"mimetype": "audio/mp4",
"bytes": 4887996,
"file_modification": "2024-12-04T00:04:55.328Z",
"av_metadata": {
"format": "MPEG-4",
"codec_id": "M4A ",
"duration": 297.13,
"audio_count": 1,
"file_extension": "m4a",
"format_profile": "Apple audio with iTunes info"
},
"dro_file_parts": [
{
"part_type": "audio",
"part_id": "1",
"order": 0,
"format": "AAC",
"audio_metadata": {
"channels": "2",
"codec_id": "mp4a-40-2",
"stream_size": 4831404,
"sampling_rate": 48000,
"mean_volume": -35,
"max_volume": -25
}
}
]
},
{
"druid": "druid:bc123df4567",
"filename": "file1.mp4",
"filetype": "fmt/199",
"mimetype": "video/mp4",
"bytes": 4887996,
"file_modification": "2024-12-04T00:04:55.328Z",
"av_metadata": {
"format": "MPEG-4",
"codec_id": "M4A ",
"duration": 297.13,
"audio_count": 1,
"file_extension": "m4a",
"format_profile": "Apple audio with iTunes info"
},
"dro_file_parts": [
{
"part_type": "audio",
"part_id": "1",
"order": 0,
"format": "AAC",
"audio_metadata": {
"channels": "2",
"codec_id": "mp4a-40-2",
"stream_size": 4831404,
"sampling_rate": 48000,
"mean_volume": -35,
"max_volume": -25
}
}
]
},
{
"druid": "druid:bc123df4567",
"filename": "file2.mp4",
"filetype": "fmt/199",
"mimetype": "video/mp4",
"bytes": 4887996,
"file_modification": "2024-12-04T00:04:55.328Z",
"av_metadata": {
"format": "MPEG-4",
"codec_id": "M4A ",
"duration": 297.13,
"audio_count": 1,
"file_extension": "m4a",
"format_profile": "Apple audio with iTunes info"
},
"dro_file_parts": [
{
"part_type": "audio",
"part_id": "1",
"order": 0,
"format": "AAC",
"audio_metadata": {
"channels": "2",
"codec_id": "mp4a-40-2",
"stream_size": 4831404,
"sampling_rate": 48000,
"mean_volume": -35,
"max_volume": -25
}
}
]
},
{
"druid": "druid:bc123df4567",
"filename": "file3.mp4",
"filetype": "fmt/199",
"mimetype": "video/mp4",
"bytes": 4887996,
"file_modification": "2024-12-04T00:04:55.328Z",
"av_metadata": {
"format": "MPEG-4",
"codec_id": "M4A ",
"duration": 297.13,
"audio_count": 1,
"file_extension": "m4a",
"format_profile": "Apple audio with iTunes info"
},
"dro_file_parts": [
{
"part_type": "audio",
"part_id": "1",
"order": 0,
"format": "AAC",
"audio_metadata": {
"channels": "2",
"codec_id": "mp4a-40-2",
"stream_size": 4831404,
"sampling_rate": 48000,
"mean_volume": -35,
"max_volume": -25
}
}
]
},
{
"druid": "druid:bc123df4567",
"filename": "file1.txt",
"filetype": "fmt/817",
"mimetype": "application/text",
"bytes": 88457,
"file_modification": "2024-12-04T00:13:16.502Z"
},
{
"druid": "druid:bc123df4567",
"filename": "file2.txt",
"filetype": "fmt/817",
"mimetype": "application/text",
"bytes": 91457,
"file_modification": "2024-12-04T00:13:16.502Z"
}
]
144 changes: 144 additions & 0 deletions spec/fixtures/technical_metadata/bc123df4567_no_audio_track.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
[
{
"druid": "druid:bc123df4567",
"filename": "file1.m4a",
"filetype": "fmt/199",
"mimetype": "audio/mp4",
"bytes": 4887996,
"file_modification": "2024-12-04T00:04:55.328Z",
"av_metadata": {
"format": "MPEG-4",
"codec_id": "M4A ",
"duration": 0,
"audio_count": 0,
"file_extension": "m4a",
"format_profile": "Apple audio with iTunes info"
},
"dro_file_parts": [
{
"part_type": "audio",
"part_id": "1",
"order": 0,
"format": "AAC",
"audio_metadata": {
"channels": "2",
"codec_id": "mp4a-40-2",
"stream_size": 4831404,
"sampling_rate": 48000
}
}
]
},
{
"druid": "druid:bc123df4567",
"filename": "file1.mp4",
"filetype": "fmt/199",
"mimetype": "video/mp4",
"bytes": 4887996,
"file_modification": "2024-12-04T00:04:55.328Z",
"av_metadata": {
"format": "MPEG-4",
"codec_id": "M4A ",
"duration": 297.13,
"audio_count": 1,
"file_extension": "m4a",
"format_profile": "Apple audio with iTunes info"
},
"dro_file_parts": [
{
"part_type": "audio",
"part_id": "1",
"order": 0,
"format": "AAC",
"audio_metadata": {
"channels": "2",
"codec_id": "mp4a-40-2",
"stream_size": 4831404,
"sampling_rate": 48000,
"mean_volume": -35,
"max_volume": -25
}
}
]
},
{
"druid": "druid:bc123df4567",
"filename": "file2.mp4",
"filetype": "fmt/199",
"mimetype": "video/mp4",
"bytes": 4887996,
"file_modification": "2024-12-04T00:04:55.328Z",
"av_metadata": {
"format": "MPEG-4",
"codec_id": "M4A ",
"duration": 297.13,
"audio_count": 1,
"file_extension": "m4a",
"format_profile": "Apple audio with iTunes info"
},
"dro_file_parts": [
{
"part_type": "audio",
"part_id": "1",
"order": 0,
"format": "AAC",
"audio_metadata": {
"channels": "2",
"codec_id": "mp4a-40-2",
"stream_size": 4831404,
"sampling_rate": 48000,
"mean_volume": -35,
"max_volume": -25
}
}
]
},
{
"druid": "druid:bc123df4567",
"filename": "file3.mp4",
"filetype": "fmt/199",
"mimetype": "video/mp4",
"bytes": 4887996,
"file_modification": "2024-12-04T00:04:55.328Z",
"av_metadata": {
"format": "MPEG-4",
"codec_id": "M4A ",
"duration": 297.13,
"audio_count": 1,
"file_extension": "m4a",
"format_profile": "Apple audio with iTunes info"
},
"dro_file_parts": [
{
"part_type": "audio",
"part_id": "1",
"order": 0,
"format": "AAC",
"audio_metadata": {
"channels": "2",
"codec_id": "mp4a-40-2",
"stream_size": 4831404,
"sampling_rate": 48000,
"mean_volume": -35,
"max_volume": -25
}
}
]
},
{
"druid": "druid:bc123df4567",
"filename": "file1.txt",
"filetype": "fmt/817",
"mimetype": "application/text",
"bytes": 88457,
"file_modification": "2024-12-04T00:13:16.502Z"
},
{
"druid": "druid:bc123df4567",
"filename": "file2.txt",
"filetype": "fmt/817",
"mimetype": "application/text",
"bytes": 91457,
"file_modification": "2024-12-04T00:13:16.502Z"
}
]
Loading