Kx Systems is, interestingly, an Irish company.
The tick data for a year takes up about 1 Tb of data. The Kx people
commented that that you can buy tick data on CD/DVDs from the NYSE.
As it turns out, this doesn't seem to be true. But instraday tick data is
available from sources like Tick Data. Buying data from a vendor is
probably the fastest way to get data into the system. Load data
and then update the data with more recent data from a
historical data feed. Then when you have a real time data feed,
stream the data both into the system and to the trading models.
The Kx software pricing is pretty eye popping. On the other hand,
it's cheap compared to developing the same functionality yourself.
They said that there are two models, one being a subscription model,
which is about half the cost of buying a software license. So the
software license is $100,000 and 20% per year and the subscription
model is $50,000 per year.
I also asked them about the kind of systems that one should get to
run the Kx software. It would appear that you need two things:
- Fast disk
- Lots of memory.
For a disk system one would want a fast RAID with something like 2
terabytes of storage. For memory, as you suggested, 32 Gb of
memory. I assume that they use the memory to hide the disk latency
(e.g., they buffer a lot of data in memory).
Along with the cost of Kx is the learning curve when it comes to
setting up the system and learning to use it.
If you write software to create your own trading system, it's not
clear to me if you need something like Kx. A flat file system could
be created with a directory for each year and then subdirectories for
each month. Within the month directory there would be directories for
each day. The day directories would have the tick data files for each stock.