Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix general observations fmuobs for dictionaries #648

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
a503cbc
Add full suite of drogon test data
daniel-sol Jan 4, 2024
f056de5
Make one drogon obs file
daniel-sol Jan 5, 2024
11d1e60
Add test data specifically for general observations
daniel-sol Jan 5, 2024
17c18e7
Introduce parent_dir in function
daniel-sol Jan 5, 2024
a88a034
Add log statement if no includes
daniel-sol Jan 5, 2024
165577b
Add reader for general observation files
daniel-sol Jan 5, 2024
9bb9646
Introduce parent_dir in function
daniel-sol Jan 5, 2024
2e42d6f
Introduce reading of general observations
daniel-sol Jan 5, 2024
378ebf6
Align test with drogon changes
daniel-sol Jan 5, 2024
721a3fe
Add assert messages
daniel-sol Jan 5, 2024
2abd96d
Focus tests around general_df2obsdict
daniel-sol Jan 5, 2024
36f6790
Add drogon_tracer for general_df2obsdict
daniel-sol Jan 5, 2024
2e0db6d
Modify test data to allow for general observations
daniel-sol Jan 8, 2024
e37ebaf
Add more keys for general observations
daniel-sol Jan 8, 2024
bbe8867
Remove writing of test data
daniel-sol Jan 8, 2024
be4fc35
Rename key
daniel-sol Jan 8, 2024
cac2a8d
Broaden test of general_df2obsdict
daniel-sol Jan 8, 2024
39f0854
Add functionality for reading twin file
daniel-sol Jan 9, 2024
b721887
Fix only one key in results
daniel-sol Jan 9, 2024
841ab1f
Add more specific test data for general_df2obsdict
daniel-sol Jan 9, 2024
e285ca4
Rename readers to gen_obs_writers
daniel-sol Jan 9, 2024
992c6de
Move assignment of extra keys to right level
daniel-sol Jan 9, 2024
77e9a02
Tidy test data
daniel-sol Jan 9, 2024
d677c0b
Give tests more informative name
daniel-sol Jan 9, 2024
67a333f
Add first rough sketch of adding well name and date for rft
daniel-sol Jan 9, 2024
9b9b2d3
Fix wrong spelling
daniel-sol Jan 10, 2024
722f20a
Improve length check of keys, rename function
daniel-sol Jan 10, 2024
80f90a0
Align csv file for correct results wit code changes
daniel-sol Jan 10, 2024
175d8fa
Remove print statements
daniel-sol Jan 10, 2024
12e1970
Align yml file for correct results with code changes
daniel-sol Jan 10, 2024
5826ff7
Change test data
daniel-sol Jan 10, 2024
d470ce7
Add proper unit tests for gen_obs_writers
daniel-sol Jan 11, 2024
908feaa
Add test data for Snorre
daniel-sol Jan 11, 2024
f154407
Add test_fmuobs_w_field_data.py, which is replacement for test_fmuobs…
daniel-sol Jan 11, 2024
d60f33d
Correct docstring
daniel-sol Jan 11, 2024
23d85f4
Remove H:M:S
daniel-sol Jan 11, 2024
43ab8ee
Utilize standard ert date formats and fix tests
daniel-sol Jan 11, 2024
6e86b1c
Fix logger name
daniel-sol Jan 11, 2024
8e0d2c1
Add specific function for finding well file
daniel-sol Jan 11, 2024
738f26a
Give better warning
daniel-sol Jan 11, 2024
6ff3c10
Make test for find_well_file_path
daniel-sol Jan 11, 2024
f93729f
Fix index error
daniel-sol Jan 11, 2024
e3ed95f
Track parent path for observation files
daniel-sol Jan 11, 2024
f314e5c
Add removal of RFT_ in pattern matching
daniel-sol Jan 11, 2024
dd1a754
Split out common functions for testing of fmuobs
daniel-sol Jan 11, 2024
db42d01
Swap to using common functions for general observations
daniel-sol Jan 11, 2024
56ba318
Add correct answers for drogon and snorre
daniel-sol Jan 11, 2024
329ad31
Remove timestamps in correct answer
daniel-sol Jan 11, 2024
5078bc9
Improve tests
daniel-sol Jan 11, 2024
202d4f1
Give better name of test file
daniel-sol Jan 11, 2024
e3f318c
Add test data from welltests
daniel-sol Jan 11, 2024
4ff8be4
Update ert-doc data to be in line with index_list
daniel-sol Jan 12, 2024
6d8df66
Update common function to give more detailed info
daniel-sol Jan 12, 2024
d7ee33a
Add ert-doc test data to tests
daniel-sol Jan 12, 2024
4ccc6b4
Add assignment of index_list to general_obs
daniel-sol Jan 12, 2024
7d2460b
Add function to tidy keys for general observations
daniel-sol Jan 12, 2024
0eef81c
Fix errors caused by intro of tidy_general_obs_dict
daniel-sol Jan 12, 2024
08abc49
Bring linting up to repo standards
daniel-sol Jan 12, 2024
4086150
Ignore type error
daniel-sol Jan 15, 2024
a145cb4
Remove unused imports
daniel-sol Jan 15, 2024
298c45b
correct wrong assignment by swapping md and z
daniel-sol Jan 15, 2024
a8f1cff
Rename observations to value
daniel-sol Jan 16, 2024
af6c6f7
Align with existing format for summary
daniel-sol Jan 17, 2024
cdd6a67
Adapt test data to breaking changes
daniel-sol Jan 17, 2024
33fc2a2
Correct grammar
daniel-sol Jan 17, 2024
655c675
Remove dump of temp file
daniel-sol Jan 17, 2024
f87dca9
Align tests with latest
daniel-sol Jan 17, 2024
6cf15a0
Use more neutral name
daniel-sol Jan 17, 2024
c2a0792
Remove welltest test data
daniel-sol Jan 19, 2024
493316b
Convert relative path to absolute path
daniel-sol Jan 19, 2024
48e956e
Add option for renaming
daniel-sol Jan 26, 2024
c04d1db
Ensure tests runs
daniel-sol Jan 26, 2024
6a4c3a0
Isort
daniel-sol Jan 29, 2024
f791d28
Fix test data
daniel-sol Jan 29, 2024
25f7d99
Black
daniel-sol Jan 30, 2024
e0b0467
Tidy test data and test
daniel-sol Jan 30, 2024
25718a9
Convert set into list
daniel-sol Jan 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/subscript/bjobsusers/bjobsusers.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,9 @@ def get_jobs(status: str, bjobs_function: Callable) -> pd.DataFrame:
data = [
[
uname,
1
if rex.match(hname) is None
else int(rex.match(hname).group(1)), # type: ignore
(
1 if rex.match(hname) is None else int(rex.match(hname).group(1))
), # type: ignore
]
for (uname, hname) in slines
]
Expand Down
24 changes: 12 additions & 12 deletions src/subscript/casegen_upcars/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -470,9 +470,9 @@ def _build_grid(self):
if self._a * self._b * self._c == 0.0:
self.dict_info["ModelDescription"] = f"Slab with tilting angle {self._tilt}"
else:
self.dict_info[
"ModelDescription"
] = "Dome structure with radius (x, y, z) : {:.2f}m, {:.2f}m, {:.2f}m"
self.dict_info["ModelDescription"] = (
"Dome structure with radius (x, y, z) : {:.2f}m, {:.2f}m, {:.2f}m"
)

x_mid = self._centroid_x * self._lx
y_mid = self._centroid_y * self._ly
Expand All @@ -494,13 +494,13 @@ def _build_grid(self):
cell_dz = np.empty((self._total_nx, self._total_ny, self._total_nz))

for idx in self._fracture_i:
cell_dx[
idx : idx + self._fracture_cell_count, :, :
] = self._fracture_thickness
cell_dx[idx : idx + self._fracture_cell_count, :, :] = (
self._fracture_thickness
)
for idx in self._fracture_j:
cell_dy[
:, idx : idx + self._fracture_cell_count, :
] = self._fracture_thickness
cell_dy[:, idx : idx + self._fracture_cell_count, :] = (
self._fracture_thickness
)
for idx in range(self._nz):
cell_dz[:, :, idx] = self._layer_dz[idx]
self._cell_volume = cell_dx * cell_dy * cell_dz
Expand Down Expand Up @@ -779,9 +779,9 @@ def export_grdecl(self, filename):
)

for idx in range(1, self._total_nz):
zcorn[
n_surface_points * 2 * idx : n_surface_points * (2 * idx + 1)
] = zcorn[n_surface_points * (2 * idx - 1) : n_surface_points * (2 * idx)]
zcorn[n_surface_points * 2 * idx : n_surface_points * (2 * idx + 1)] = (
zcorn[n_surface_points * (2 * idx - 1) : n_surface_points * (2 * idx)]
)
zcorn[
n_surface_points * (2 * idx + 1) : 2 * n_surface_points * (idx + 1)
] = (
Expand Down
1 change: 1 addition & 0 deletions src/subscript/casegen_upcars/udf.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
""" Supporting functions/class for casegen_upcars """

import numpy as np
from scipy.stats import uniform

Expand Down
7 changes: 4 additions & 3 deletions src/subscript/check_swatinit/check_swatinit.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
"""SWATINIT qc tool"""

import argparse
import sys
from typing import Any, Dict, List
Expand Down Expand Up @@ -356,9 +357,9 @@ def qc_flag(qc_frame: pd.DataFrame) -> pd.DataFrame:
# Below a nonzero capillary entry pressure but above the contact,
# SWAT and SWATINIT should be 1.
if "PPCW" in qc_frame:
qc_col[
np.isclose(qc_frame["SWAT"], 1) & np.isclose(qc_frame["PPCW"], 0)
] = __PC_SCALED__
qc_col[np.isclose(qc_frame["SWAT"], 1) & np.isclose(qc_frame["PPCW"], 0)] = (
__PC_SCALED__
)

# PC is scaled (including "scaling" using PC_SCALING=1), but
# SWAT!=SWATINIT, this must be due to EQUIL item 9 being nonzero.
Expand Down
1 change: 1 addition & 0 deletions src/subscript/csv_stack/csv_stack.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
"""Tool to stack/pivot CSV files, exposed as command line client,
ERT workflow and ERT forward model"""

import argparse
import logging
import re
Expand Down
1 change: 1 addition & 0 deletions src/subscript/fmu_copy_revision/fmu_copy_revision.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
Purpose is fast and secure copy for fmu revisions using rsync as engine,
with xargs to speed up multithreading.
"""

import argparse
import getpass
import shutil
Expand Down
10 changes: 6 additions & 4 deletions src/subscript/fmuobs/fmuobs.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
"""fmuobs is a converter tool for observation files used in assisted
history matching"""

import argparse
import logging
import os
import signal
import sys
from pathlib import Path
from pathlib import Path, PosixPath
from typing import Optional, Tuple, Union

import pandas as pd
Expand Down Expand Up @@ -362,8 +363,8 @@ def fmuobs(
f"Unsupported ERROR_MODE : {', '.join(error_mode)}. "
"Please verify the output file"
)

dump_results(dframe, csv, yml, resinsight, ertobs)
logger.debug("After all the hard work, what is returned is this %s", dframe)
dump_results(dframe, csv, yml, resinsight, ertobs, PosixPath(inputfile).parent)


def dump_results(
Expand All @@ -372,6 +373,7 @@ def dump_results(
yamlfile: Optional[str] = None,
resinsightfile: Optional[str] = None,
ertfile: Optional[str] = None,
parent_dir: PosixPath = PosixPath("."),
) -> None:
"""Dump dataframe with ERT observations to CSV and/or YML
format to disk. Writes to stdout if filenames are "-". Skips
Expand All @@ -397,7 +399,7 @@ def dump_results(
dframe.to_csv(sys.stdout, index=False)

if yamlfile:
obs_dict_for_yaml = df2obsdict(dframe)
obs_dict_for_yaml = df2obsdict(dframe, parent_dir)
if not obs_dict_for_yaml and not dframe.empty:
logger.error("None of your observations are supported in YAML")
yaml_str = yaml.safe_dump(obs_dict_for_yaml)
Expand Down
Loading