Support Arrow Tensor type

[eddyxu]

edit: use arrow::extension::FixedShapeTensorType

https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html#fixed-shape-tensor-extension

Problem

In arrow, Tensors do not have corresponding Array type, as it can not be used to store in a Table / Dataset / RecordBatch.

To support ML datasets, it is desirable to have tensors stored within the datasets.

Desired Behavior

To support parametrized Tensors in a Table / Datasets. A TensorArray and TensorType should allow parameters, i.e., shade, data type (i.e., bid_width) and etc.

class TensorType {
   TensorType(data_type, shade, ...)
}

[rok]

arrow::extension::FixedShapeTensorType and arrow::extension::VariableShapeTensorType will provide getting individual tensors from their respective arrays. arrow::extension::FixedShapeTensorArray provides to_numpy_ndarray on sliced array allowing for batch-sized tensors. I think this covers our desired use case.
Am I missing something or can we close this ticket?

[rok]

@eddyxu

[judahrand]

https://github.com/apache/arrow/blob/5b9f4b976118052824c5aeaa2220886839b6f4be/cpp/src/arrow/extension/fixed_shape_tensor.h#L23