Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audio upload handler #7113

Closed
wants to merge 68 commits into from
Closed
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
b428863
Audio upload handler
kushalpoddar Nov 8, 2023
0551bde
Added transcript field
kushalpoddar Jan 27, 2024
4135657
Saving Auto Annotation
kushalpoddar Mar 4, 2024
1f1cdd7
mimicking the request
kushalpoddar Mar 8, 2024
746e44e
fix: ai annotation url changed
rohan220217 Mar 9, 2024
30bd9bf
Transcript not saving bug fixed
kushalpoddar Mar 11, 2024
2f5c7f3
Update production.py
kushalpoddar Mar 13, 2024
ce4ab15
Export functionality
kushalpoddar Apr 2, 2024
8550ed2
Modified Nginx /data path
kushalpoddar Apr 2, 2024
e9b7961
Merge branch 'feat/audino-v2' of https://github.com/kushalpoddar/cvat…
kushalpoddar Apr 2, 2024
678afc7
Added functionality for gender, age and locale for audios
kushalpoddar Apr 5, 2024
230b28b
bug fixes
kushalpoddar Apr 6, 2024
7223610
Added accent and emotion in annotation
kushalpoddar Apr 6, 2024
b0c1335
Migrations for accent and emotion
kushalpoddar Apr 6, 2024
a9a53e4
Email settings: wip
kushalpoddar Apr 7, 2024
ab36c8a
Added functionality for diff export formats
kushalpoddar Apr 9, 2024
20223f4
fix
kushalpoddar Apr 9, 2024
d188ab1
Status API done
kushalpoddar Apr 15, 2024
cf97b3b
wip
kushalpoddar Apr 15, 2024
7912982
bug fix
kushalpoddar Apr 15, 2024
b933f1d
Ai auto annotation bug fix
kushalpoddar Apr 16, 2024
e773d31
File upload status fixed
kushalpoddar Apr 16, 2024
0e42dae
bug fix
kushalpoddar Apr 20, 2024
6589411
Email templating changes
kushalpoddar Apr 26, 2024
2d29065
Added pydub
kushalpoddar May 6, 2024
2da6517
alter col
kushalpoddar May 6, 2024
74e6743
added new audio total durstion field
kushalpoddar May 6, 2024
6a602b6
Updated migrations
kushalpoddar May 8, 2024
97a158b
Audio duration finding using av
kushalpoddar May 9, 2024
bfaebcc
local
kushalpoddar May 9, 2024
531de53
Added UI Url
kushalpoddar May 9, 2024
44df236
Bug fix
kushalpoddar May 9, 2024
c696325
Bug fix
kushalpoddar May 10, 2024
5c8895a
readded pydub
kushalpoddar May 10, 2024
0799c54
bug fix for av
kushalpoddar May 10, 2024
9991d03
Voxpopuli bug fix and segment size rechanged
kushalpoddar May 21, 2024
16e0509
added email notification when annotation is done
ashish7515 May 26, 2024
f3beafa
Bug fixes
kushalpoddar May 29, 2024
111d2b6
resolve bug for large files
ashish7515 May 30, 2024
32a065b
merge conflicts
ashish7515 May 30, 2024
0bde3ff
update
ashish7515 May 30, 2024
62e94bb
bug fix
kushalpoddar May 30, 2024
0ff4e30
Encoding bugresolved for audio
kushalpoddar May 30, 2024
4cc294c
Added chardet to requirements
kushalpoddar May 30, 2024
e2b4c07
bug fix for audios
kushalpoddar May 30, 2024
9b7d1ac
Merge pull request #2 from midas-research/my-branch
rohan220217 Jun 3, 2024
030e245
start end in download csv, mp3 format, typo error
ashish7515 Jun 5, 2024
ebe46a5
comment removed
ashish7515 Jun 6, 2024
2cf5d5a
Merge pull request #3 from midas-research/my-branch
rohan220217 Jun 6, 2024
a52c010
Merge branch 'feat/audino-v2' of https://github.com/midas-research/cv…
ashish7515 Jun 15, 2024
f23cd60
conflict resolved
ashish7515 Jun 15, 2024
a628c20
Merge pull request #4 from midas-research/feat/ground_truth
rohan220217 Jun 15, 2024
95253ff
rebased
kushalpoddar Jun 16, 2024
6779ccf
Bug fixes
kushalpoddar Jun 16, 2024
90d8d08
fix: quality report bug
ashish7515 Jun 19, 2024
66733ca
Merge pull request #6 from midas-research/feat/ground_truth
rohan220217 Jun 19, 2024
7a86c95
Add: settings parameter from user
ashish7515 Jun 22, 2024
ac9f840
Revert "Add: settings parameter from user"
ashish7515 Jun 22, 2024
d1e3411
"Add: settings parameter from user"
ashish7515 Jun 22, 2024
ab6c2ef
Add: comparing extra parameter check
ashish7515 Jun 22, 2024
2c46ad8
Fix: quality settings
ashish7515 Jun 24, 2024
ecc54b4
Merge pull request #7 from midas-research/feat/ground_truth
rohan220217 Jun 27, 2024
b9a2fc6
Fix: keyerror in analytics
ashish7515 Jun 27, 2024
1dbdfeb
Merge pull request #8 from midas-research/feat/ground_truth
rohan220217 Jun 30, 2024
025d5aa
fix: update parameters threshold value
ashish7515 Jul 2, 2024
5f2d84c
Merge pull request #9 from midas-research/feat/ground_truth
rohan220217 Jul 2, 2024
2340bd3
fix: export annotation bug when no attributes present
ashish7515 Jul 3, 2024
7f6387c
Merge pull request #10 from midas-research/feat/ground_truth
rohan220217 Jul 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ source =
cvat/apps/
cvat-sdk/
cvat-cli/
utils/dataset_manifest
utilsset_manifest

omit =
cvat/settings/*
Expand Down
16 changes: 1 addition & 15 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -167,9 +167,7 @@
"env": {
"CVAT_SERVERLESS": "1",
"ALLOWED_HOSTS": "*",
"IAM_OPA_BUNDLE": "1",
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
"IAM_OPA_BUNDLE": "1"
},
"args": [
"runserver",
Expand Down Expand Up @@ -211,8 +209,6 @@
"django": true,
"cwd": "${workspaceFolder}",
"env": {
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
},
"console": "internalConsole"
},
Expand All @@ -233,8 +229,6 @@
"django": true,
"cwd": "${workspaceFolder}",
"env": {
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
},
"console": "internalConsole"
},
Expand All @@ -255,8 +249,6 @@
"django": true,
"cwd": "${workspaceFolder}",
"env": {
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
},
"console": "internalConsole"
},
Expand All @@ -277,8 +269,6 @@
"django": true,
"cwd": "${workspaceFolder}",
"env": {
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
},
"console": "internalConsole"
},
Expand All @@ -296,8 +286,6 @@
"-i", "1"
],
"env": {
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
},
"console": "internalConsole"
},
Expand Down Expand Up @@ -356,8 +344,6 @@
"django": true,
"cwd": "${workspaceFolder}",
"env": {
"DJANGO_LOG_SERVER_HOST": "localhost",
"DJANGO_LOG_SERVER_PORT": "8282"
},
"console": "internalConsole"
},
Expand Down
62 changes: 62 additions & 0 deletions cvat/apps/dataset_manager/formats/aud_common_voice.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Copyright (C) 2024 CVAT.ai Corporation
#
# SPDX-License-Identifier: MIT

# Copyright (C) 2019-2022 Intel Corporation
# Copyright (C) 2023 CVAT.ai Corporation
#
# SPDX-License-Identifier: MIT

import os.path as osp
from glob import glob

from pyunpack import Archive

from cvat.apps.dataset_manager.bindings import (GetCVATDataExtractor,
import_dm_annotations, match_dm_item, find_dataset_root)
from cvat.apps.dataset_manager.util import make_zip_archive
from datumaro.components.extractor import DatasetItem
from datumaro.components.project import Dataset
from datumaro.plugins.yolo_format.extractor import YoloExtractor

from .registry import dm_env, exporter, importer
from cvat.apps.engine.log import ServerLogManager
slogger = ServerLogManager(__name__)


@exporter(name='YOLO', ext='ZIP', version='1.1')
def _export(dst_file, temp_dir, instance_data, save_images=False):
slogger.glob.info("HEYLO YOLO EXPORTER AUDINO")
# slogger.glob.debug()
dataset = Dataset.from_extractors(GetCVATDataExtractor(
instance_data, include_images=save_images), env=dm_env)

dataset.export(temp_dir, 'yolo', save_images=save_images)

make_zip_archive(temp_dir, dst_file)

Comment on lines +27 to +37
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider renaming _export to reflect its specific use for YOLO format or add a comment clarifying its purpose.

@importer(name='YOLO', ext='ZIP', version='1.1')
def _import(src_file, temp_dir, instance_data, load_data_callback=None, **kwargs):
Archive(src_file.name).extractall(temp_dir)

image_info = {}
frames = [YoloExtractor.name_from_path(osp.relpath(p, temp_dir))
for p in glob(osp.join(temp_dir, '**', '*.txt'), recursive=True)]
root_hint = find_dataset_root(
[DatasetItem(id=frame) for frame in frames], instance_data)
for frame in frames:
frame_info = None
try:
frame_id = match_dm_item(DatasetItem(id=frame), instance_data,
root_hint=root_hint)
frame_info = instance_data.frame_info[frame_id]
except Exception: # nosec
pass
if frame_info is not None:
image_info[frame] = (frame_info['height'], frame_info['width'])

dataset = Dataset.import_from(temp_dir, 'yolo',
env=dm_env, image_info=image_info)
if load_data_callback is not None:
load_data_callback(dataset, instance_data)
import_dm_annotations(dataset, instance_data)
Comment on lines +38 to +62
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding error logging in the exception handling block to aid in debugging.

4 changes: 3 additions & 1 deletion cvat/apps/dataset_manager/formats/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ def make_exporter(name):
import cvat.apps.dataset_manager.formats.mots
import cvat.apps.dataset_manager.formats.pascal_voc
import cvat.apps.dataset_manager.formats.tfrecord
import cvat.apps.dataset_manager.formats.yolo
# import cvat.apps.dataset_manager.formats.yolo
import cvat.apps.dataset_manager.formats.imagenet
import cvat.apps.dataset_manager.formats.camvid
import cvat.apps.dataset_manager.formats.widerface
Expand All @@ -125,3 +125,5 @@ def make_exporter(name):
import cvat.apps.dataset_manager.formats.lfw
import cvat.apps.dataset_manager.formats.cityscapes
import cvat.apps.dataset_manager.formats.openimages
#Audino Export Formats
import cvat.apps.dataset_manager.formats.aud_common_voice
3 changes: 3 additions & 0 deletions cvat/apps/dataset_manager/formats/yolo.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,13 @@
from datumaro.plugins.yolo_format.extractor import YoloExtractor

from .registry import dm_env, exporter, importer
from cvat.apps.engine.log import ServerLogManager
slogger = ServerLogManager(__name__)


@exporter(name='YOLO', ext='ZIP', version='1.1')
def _export(dst_file, temp_dir, instance_data, save_images=False):
slogger.glob.info("HEYLO YOLO EXPORTER")
Comment on lines +19 to +25
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introduction of logging in YOLO export function is good for traceability. Consider refining the log message to be more descriptive of the operation being performed.

- slogger.glob.info("HEYLO YOLO EXPORTER")
+ slogger.glob.info("Starting export of YOLO dataset")

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
from cvat.apps.engine.log import ServerLogManager
slogger = ServerLogManager(__name__)
@exporter(name='YOLO', ext='ZIP', version='1.1')
def _export(dst_file, temp_dir, instance_data, save_images=False):
slogger.glob.info("HEYLO YOLO EXPORTER")
from cvat.apps.engine.log import ServerLogManager
slogger = ServerLogManager(__name__)
@exporter(name='YOLO', ext='ZIP', version='1.1')
def _export(dst_file, temp_dir, instance_data, save_images=False):
slogger.glob.info("Starting export of YOLO dataset")

dataset = Dataset.from_extractors(GetCVATDataExtractor(
instance_data, include_images=save_images), env=dm_env)

Expand Down
Loading