Working with large amount of feta fetching from websocket

Gonna need some help here. I'm trying to code a program in Rails 7 that would fetch data from Binance's websocket (specifically, all klines data for all symbols and intervals). Problem is that, considering the extremely large amount of data being fetched literally every 2 seconds, the server doesn't handle it correctly and I end up with large gaps. I don't really know how to go around this. What I'm doing is creating the connections to the websocket for all the necessary streams (around 4000), store it in a queue then do a raw insert into a temporary table with CSV then transfer it into a final table. How can I work around this?

>>38444 So gaps are because you treat your 4000 streams sequentially and load from each of 4000 websocket in queue? Like first BTC - then after no lement you take ETH from queue and then after no elements in ETH you take another currency stream from queue?? IMO there is more specialized tech for doing that. You can read all streams in parallel. Do you know what Spark is? in Spark Streaming you just do "someStream = spark.readStream(websocketURL)..." and then redirect it to output. Spark with Pyspark is not that hard to install locally if you have some spare credits you can create Databricks instance on Azure which is easiest way to work with Spark. Do not save streams into CSV. Instead you should use parquet or better - Delta format which has build-in history of changes to file - so it wont get corrupted. Delta/parquet format is default way of writing data in Spark which is build for efficient handling of large amounts of data. Entire point is that you can "partition" data on some "column" and it will save correctly your data to correct directory structure - basically filesystem as a database. Then all querying will be fast because system will know in which directory is data for which parition and will automatically skip all other data.

>>38492 >lain talking spark Gonna set up a chair and watch in the corner. Don't mind me.