Reading in chicago_crime_2019.mat file

[tgkolda]

I'm trying to figure out how to read in the chicago_crime_2019.mat file from https://gitlab.com/tensors/tensor_data_chicago_crime.

Here's what I have so far:

from scipy.io import loadmat

# Load the MATLAB file
mat_file_path = "C:\\Users\\TammyKolda\\Documents\\GIT Repos\\tensor_data_chicago_crime\\chicago_crime_2019.mat"
mat_data = loadmat(mat_file_path)

# Display the keys in the MATLAB file to understand its structure
print(mat_data.keys())

# Extract the 'X' object from mat_data
X_mat = mat_data['X']

# Extract the 'subs', 'vals', and 'size' arrays from the 'X' object
subs = X_mat['subs'][0, 0]
vals = X_mat['vals'][0, 0]
size = X_mat['size'][0, 0]

# Adjust subs to be zero-indexed
subs_zero_indexed = subs - 1

# Create the sparse tensor
X = ttb.sptensor.from_aggregator(subs_zero_indexed, vals, (365,24,77,12))

One thing is that I had to type in the size manually. I guess this is not a numpy array? How can I use the size that was read in?

[dmdunla]

I think that we should consider creating “from_mat” methods in the data classes, or create a utility called “import_mat” akin to “import_data”. The former could be tailored to each data classes and associated error checking may be helpful. The latter could become unwieldy in trying to support all data classes.

Malia, since there are two versions of MATLAB *.mat files currently used in practice as far as I see, supporting imports using scipy or h5py may be better handled within pyttb rather than relying on users to find info in tutorials.

[ntjohnson1]

You called the aggregator before so I included both. The default constructor will just take the data as is. The aggregator will merge duplicates subs (which you probably don't need since you exported from matlab)

[ntjohnson1]

When I loaded that dataset the size from matlab was a 2D numpy row vector. So until we make our shape arguments more flexible that was a sufficient transform for me (and I expect that to work across mat files for the size to shape conversion based on our row vector discussion elsewhere)

Elaborating in case it helps, flatten goes from [[1,2,3]] to [1,2,3] then we cast to tuple. Direct cast yields a single item tuple containing a 1D array.

[tgkolda]

This construction is different than MATLAB. MATLAB by default does aggregation. Then there are special hidden ways to call that skip the aggregation. This isn't advertised because it's unsafe. Is there a reason that this is different? I would imagine the default could be to make it so that sparse = ttb.sptensor(subs, vals, shape=tuple(shape.flatten())) does aggregation.

[ntjohnson1]

I think we wanted to make it more distinctive which construction was being used. Since the aggregation takes more time opting in rather than out seemed reasonable. We add a note that minimal checking is done via direct construction, and reference the aggregation if people aren't pre-processing their data.

Not resolving duplicates provided explicitly also aligns with scipy's COO if I remember correctly.

[tgkolda]

There can be major downstream problems with a failure to aggregate, so I would recommend against this being the default. Time to build is a one-time expense and rarely a major factor. I would only expect our pyttb developers to need this functionality, if at all. Better to pay the expense and have correct downstream behaviors.

[tgkolda]

I don't think there's any need to create a reader function. However, I would like to have guidance. For both sparse and dense tensors. I will experiment with dense soon. I'm not sure how the ordering will translate, and that's the main thing I want to check.

[dmdunla]

I agree with Nick about taking on the technical debt of tracking the MATLAB file format in pyttb. I suggest that we include a tutorial on importing and exporting data (I do not see anything on this right now). In other work, we have been developing workflows going from csv or binary data to pandas DataFrames to pyttb tensors, and I can see that tutorials may be a better option for users than trying to keep up with external tools in that case, too.

As for the call to from_aggregator, I agree that the default behavior could be improved. I verified that the default behavior for handling repeats can be confusing, as the repeats are propagated through the __init__, but then this could lead to undefined behavior in several of the methods.

[tgkolda]

I have figured out how to do both sparse and dense tensors in mat files, I think. It's not very hard. I would be happy to help with a tutorial of some sort, but I would need a few pointers to get started.

[ntjohnson1]

That would be great! Later this week I'll try to add some additional notes to our contributing doc about how to generate tutorials. That way we capture that for future work as well.

[ntjohnson1]

I created an issue and a draft PR with some added instructions for creating a new tutorial.

For general questions about interacting with the pyttb and getting the tutorial in place please add questions/discussions to the linked issue. Please create an issue for the tutorial(s) you're going to add and we can discuss an specifics to that there or on the PR. We have a CONTRIBUTING.md that should capture the steps/details (with the addition on my new PR specific to tutorials).