When developing software that records input signals (amounts) instantly, just how can this data be best saved and compressed? Would an SQL engine be great with this, enabling fast data mining later on, or exist other data formats that might be appropriate or compressed enough for up to 1000 data samples per second?
I do not mind building in VC++ but ideas relevant to C# could be ideal.
It's difficult to say without more information, for example, what's the source, are you requiring to question the saved data, and so forth.
However for 1000 samples/sec, you need to propably take a look at holding a couple of seconds of information in memory, after which writing them in bulk to persistent storage on another thread. (Multi-processor machine suggested).
If you choose to get it done using a handled language, keep your same data structure around to keep the samples - to ensure that the GC need not collect memory too frequently. You will get marginally better performance by utilizing pointers and also the unsafe keyword (provides direct accessibility memory structure and removes bounds checking code for arrays).
I'm not sure just how much CPU time is required that you should collect each sample and just how time-critical it's to see each sample in a specified time (can they be buffered within the device you're reading through from ?). When the sampling 's time-critical, you've 1 ms per sample and you most likely can't afford the chance of the garbage collector kicking in, because it will block your thread for a while. Within this situation, I'd choose an unmanaged approach.
SQL Server would easily have the ability to hold your computer data, or you might write these to personal files. It mostly is dependent on the thing you need related to the information at another time. I'm not sure just how much data each sample is, but let us assume it's 8 bytes. Then you've 8000 bytes per second to create of raw data - possibly you've some overhead, so it may be 10 kB/s. Most storage systems I'm able to think about will have the ability to write data only at that speed. Just make certain to create on another thread than the one which do the sampling.