Fortran versus C order

[tgkolda]

There is a lot of confusion for me about how tensors are stored and what this means for linear indexing, various efficient computations, etc. There is a note that "Note that 1D tensors (i.e., when len(shape)==1) contains a data array that follow the Numpy convention of being a row vector." in the tensor documentation. This, for instance, implies C order. Also, it seems that it is stored in C order until some other order is needed, which can be a major performance hit. I want to have a thread going as this gets sorted. It is possible to do everything in C order, but it just means revisiting a lot of things, especially some of the linear indexing, special in-place methods for TTM, etc.

[ntjohnson1]

So the PR that made things explicit about tensors being F ordered aimed to resolve some of the confusion. It made the internal storage Fortran layout almost everywhere but as you noted in the tutorial we're getting the warning about conversions occuring so obviously we need to track down where the extra conversions are occuring (which was the goal of making the loud warning to catch edge cases).

As for the 1D tensors I don't know the Numpy internals offhand but I think that for 1D arrays the layouts should be equivalent.

We have a custom implementation of linear indexing etc so if you use the tensor class access methods it should be consistent with Fortran even if the internals are inefficiently laid out in memory.

Are you seeing actual functional errors or are we just tracking down making the docs/instructions clearer?

My preference would be to make everything explicitly F ordered to start with the goal to set things up to support F and C order in the future (since C is numpys default).

As for performance we have some minimal benchmarking code setup but if we start to see issues while running real world examples we can expand that further to identify hot spots with existing tooling.

[tgkolda]

The loud warning is great and very helpful.

Are you doing both MATRICES and TENSORS in Fortran order? I ask because this may also impact some computations. I don't think it should be a major hit, but we probably need to make a list somewhere of all the functions where the layout matters.

For 1D, it is the same - C and F are the same. Nevertheless, I might suggest removing that comment as it should be irrelevant.

I'm curious about the custom implementation of linear indexing. We did our own in MATLAB because theirs didn't support conventions such as providing an array of linear indices. Is the reason here similar?

I haven't seen functional errors yet, but I also haven't been testing things at that level. I should get to that point in January.

I think it should be possible to support both F & C orders.

Is there any way to get a listing of all functions in pyttb? Both standalone functions and class functions? Thanks.

Also, what do you do in terms of performance testing? That needn't (and probably shouldn't) be a regular check-in test because it would be expensive, but it could be helpful in terms of identifying performance issues.

[ntjohnson1]

Are you doing both MATRICES and TENSORS in Fortran order? I ask because this may also impact some computations. I don't think it should be a major hit, but we probably need to make a list somewhere of all the functions where the layout matters.

By MATRICES do you mean numpy arrays or Tensor classes with two dimensions? Right now our operations assume our custom classes are internally F ordered. With the recent change I made the internal data for the dense tensors F ordered. Based on the plumbing already in place the numpy arrays returned from operations should be F ordered but we don't have extensive testing to ensure this is ALWAYS the case.

For 1D, it is the same - C and F are the same. Nevertheless, I might suggest removing that comment as it should be irrelevant.

This is mostly a note that the returned shape of the data would be (N,) instead of (N,1) rather than about the memory layout but happy to remove if confusing.

I'm curious about the custom implementation of linear indexing. We did our own in MATLAB because theirs didn't support conventions such as providing an array of linear indices. Is the reason here similar?

I think I was overly terse here. The short answer is yes but more detail just in case.

When I mentioned custom indexing I meant that fact we have an implementation of __getitem__/__setitem__ so tensor_object[idx] support the multiple flavors of indexing to match MATLAB. In numpy I believe the convention is to flatten the array then use linear indexing there (flattening just changes the view of the underylying data).

>>>import numpy as np
>>> a = np.arange(8).reshape((2,2,2))
>>>print(a)
array([[[0, 1],
        [2, 3]],
       [[4, 5],
        [6, 7]]])
>>>print(a.ravel())
array([0, 1, 2, 3, 4, 5, 6, 7])
>>>print(a.ravel()[[0,5]])
array([0, 5])

I haven't seen functional errors yet, but I also haven't been testing things at that level. I should get to that point in January.

Sounds good. We have decent test coverage, but good to keep an eye out.

I think it should be possible to support both F & C orders.

Agreed, when I made F ordering explicit I made it a parameter a variety of places to ease the transition when we decide to commit to it.

Is there any way to get a listing of all functions in pyttb? Both standalone functions and class functions? Thanks.

I think that's what the intention of our read the docs is.

Also, what do you do in terms of performance testing? That needn't (and probably shouldn't) be a regular check-in test because it would be expensive, but it could be helpful in terms of identifying performance issues.

Right now its not a regression test but more for investigation. If we come across issues in usage then we use it to diagnose where it makes sense to investigate. This is very preliminary at the moment. I think the primary focus at the moment is still to get functional coverage and bandwidth is pretty limited for that. Obviously, if people notice issues we can track them down but getting rigorous performance tests setup is probably out of scope for a while.