Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix download of qc data on deploys with local data storage #362

Merged
merged 3 commits into from
Oct 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,15 @@ Change Log
All notable changes to this project are documented in this file.


===================
21.2.1 - 2024-10-08
===================

Fixed
-----
- Fix fetching ``qc`` data from deploys with local data storage


===================
21.2.0 - 2024-07-10
===================
Expand Down
15 changes: 12 additions & 3 deletions src/resdk/tables/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import asyncio
import json
from io import BytesIO
from urllib.parse import urljoin
from urllib.parse import urljoin, urlparse

import aiohttp
import pandas as pd
Expand Down Expand Up @@ -32,16 +32,25 @@ def _uri_to_url(resolwe, uris):
return json.loads(response.content.decode("utf-8"))


def is_absolute(url: str) -> bool:
"""Return if the given URL absolute."""
return bool(urlparse(url).netloc)


async def _batch_download(resolwe, uris, parser) -> pd.DataFrame:
"""Download multiple files defined by their uri asynchronously."""
try:
uri_to_url = _uri_to_url(resolwe, uris)
except HTTPError:
return pd.DataFrame()

async with aiohttp.ClientSession() as session:
def prepare_url(url):
"""Prepent the base url if it is not absolute."""
return url if is_absolute(url) else urljoin(resolwe.url, url)

async with aiohttp.ClientSession(cookies=resolwe.session.cookies) as session:
futures = [
_download_file(uri, url, session, parser)
_download_file(uri, prepare_url(url), session, parser)
for uri, url in uri_to_url.items()
if url
]
Expand Down
3 changes: 1 addition & 2 deletions tests/functional/docs/e2e_docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -212,9 +212,8 @@ def setUp(self):
self.allow_run_process(self.res, "upload-fastq-single")
self.allow_run_process(self.res, "alignment-star")
self.allow_run_process(self.res, "workflow-bbduk-star-featurecounts-qc")
# Set permissions for using descriptor_schemas:
# Set permissions for using descriptor_schema.
self.allow_use_descriptor_schema(self.res, "reads")
self.allow_use_descriptor_schema(self.res, "sample")

def test_tutorial_create(self):
"""Test tutorial-create."""
Expand Down
Loading