Add support to http headers again

[Schwarzam]

Since version 0.4, HTTP headers are no longer passed into the reader anymore (previously contained in the storage_options), which affects access to protected data that relies on these headers (e.g., for authentication).

To address this, we can use **kwargs in the appropriate places to reintroduce header support. I’ll create a PR to implement this fix.

[delucchi-cmu]

I would rather not proliferate **storage_options back into the API. You should be able to pass the headers into the construction of a universal_pathlib.UPath object, and pass that to all methods that accept it. The headers will be handled via fsspec for all file system calls.

[Schwarzam]

Yes! This is much much better now with this new version!
I just did this below:

def get_upath(path: str | Path | UPath, **storage_options) -> UPath:
     """Returns a file pointer from a path string"""
     if not path:
         return None
     if isinstance(path, UPath):
         return path

     return UPath(path, **storage_options)

Maybe I should just rename the storage_options to something else.

[delucchi-cmu]

Ah! I meant that you can call the upath constructor in your user code instead. e.g.

from upath import UPath
import lsdb

my_path = UPath(my_path, **storage_options)
my_catalog = lsdb.read_hats(my_path)

[Schwarzam]

That's wonderful, works perfectly!
I wasn't able to think that far ahead.

Im closing the PRs