Database vs Memory Access
Nothing is faster than memory access, don’t let too young programmers or IT company managers selling their stuff say that some database is faster with a blah blah blah algorithm. Right now you can buy a motherboard with 12 Gigabytes for a reasonable price. Load the quotes into a memory array, put it into a hash table, a c# Directory, a c++ Map and “Voila” you have the fastest database in the world. Oracle and Sybase should be dead at this time …
- Is there any reason for still having a database rather than playing with memory access?
- Hummm… Yes, I see only two reasons : the data formatting and the SQL querying!
When you receive data on a file, on an IP socket, on satellite dvb channel … you never have the same dump format. At this point you have plenty of “Commercial off-the-shelf”, ready-made products for database integration. And then you can use the SQL to do your well paid job.
It is a simple question of programming a pruning and formatting tool … so I’m proud to present “prune.exe” the “Swiss-Army-Knife” for huge data dumps. With this very little tool (programmed on the Kernighan & Richie way of life) I’m albe to adapt any file to the internal format my algo testing need…
Of course, in Algorithmic trading, you only need data for testing …