Snapshot Data Conversion

[spacegoing]

I have SOD snapshot and orderbook depth update data like below:

# head -n 10 BTCUSDT_T_DEPTH_2020-07-31_depth_snap.csv 
symbol,timestamp,first_update_id,last_update_id,side,update_type,price,qty
BTCUSDT,1596153599999,45007926182,45007926182,a,snap,100000.00,0.100
BTCUSDT,1596153599999,45007926182,45007926182,a,snap,99112.00,0.225
BTCUSDT,1596153599999,45007926182,45007926182,a,snap,99059.42,0.237
BTCUSDT,1596153599999,45007926182,45007926182,a,snap,99043.00,0.001
BTCUSDT,1596153599999,45007926182,45007926182,a,snap,97654.00,0.006
BTCUSDT,1596153599999,45007926182,45007926182,a,snap,56168.22,0.060
BTCUSDT,1596153599999,45007926182,45007926182,a,snap,55918.62,0.036
BTCUSDT,1596153599999,45007926182,45007926182,a,snap,55859.72,0.085
BTCUSDT,1596153599999,45007926182,45007926182,a,snap,54732.22,0.371
# head -n 10 BTCUSDT_T_DEPTH_2020-07-31_depth_update.csv
symbol,timestamp,first_update_id,last_update_id,side,update_type,price,qty
BTCUSDT,1596153600001,45007926184,45007926184,a,set,11114.98,0.009
BTCUSDT,1596153600003,45007926187,45007926187,b,set,11111.96,4.267
BTCUSDT,1596153600003,45007926188,45007926188,a,set,11114.64,0.000
BTCUSDT,1596153600003,45007926189,45007926189,a,set,11131.95,0.686
BTCUSDT,1596153600003,45007926190,45007926190,b,set,11111.44,0.000
BTCUSDT,1596153600003,45007926191,45007926191,a,set,11114.18,0.964
BTCUSDT,1596153600003,45007926192,45007926192,a,set,11113.67,0.066
BTCUSDT,1596153600011,45007926220,45007926220,a,set,11114.98,0.000
BTCUSDT,1596153600013,45007926225,45007926225,b,set,11080.64,0.188

How do I use this with data conversion pipeline? do I still need to download trades from binance?

Many thanks!

[nkaz001]

To achieve the most accurate results, backtesting requires both the local receipt timestamp and the exchange timestamp. But, your data appears to include only one timestamp. You can still generate normalized data by creating artificial timestamps based on your assumptions. Please see https://github.com/nkaz001/hftbacktest/blob/master/py-hftbacktest/hftbacktest/data/utils/binancehistmktdata.py

[spacegoing]

@nkaz001
many tks for ur help. I created a PR for several minor bugs FYI #169

just double check if my understanding is correct:

From binance we got two types of snapshot csv files as shown above:

BTCUSDT_T_DEPTH_2020-07-31_depth_snap.csv only contains SOD depth

and

BTCUSDT_T_DEPTH_2020-07-31_depth_update.csv only contains update_type=set records

For snap.csv , we run with

sod_file = 'BTCUSDT_T_DEPTH_2020-07-01_depth_snap.csv'
sod_df = bi.convert_snapshot(sod_file, 'sod_0701.npy')

For update.csv , we run with

up_file = 'BTCUSDT_T_DEPTH_2020-07-01_depth_update.csv'
trades_file = 'BTCUSDT-trades-2020-07.csv'
up_df = bi.convert(up_file,trades_file, 'up_0701.npy')

And then run hbt with

from hftbacktest import Recorder
from hftbacktest.stats import LinearAssetRecord

asset = (
    BacktestAsset()
        .data([
            'up_0701.npy'
        ])
        .initial_snapshot('sod_0701.npy')
...
)

hbt = ROIVectorMarketDepthBacktest([asset])

[nkaz001]

You also need to set the feed_latency parameter, which is set to 0 by default, but, it is important to configure the latency appropriately. The best approach is to collect data with a local receipt timestamp from the location where your bot will run.