Data analysis
From raju
gathering data
- If you are having trouble dumping data over long horizons, dump data over small horizons and repeat. For example, instead of dumping trade data for each month, dump daily trade data and repeat for each day.
Visualizing data
- For three variables - plot correlation between two variables for different quintiles of the third variable.
Ex:- plot correlation between the Barclays Capital US High Yield Index and the S&P 500 Index for different VIX Quintiles.
delimited separated values
- Some common characters used to separate the values are comma, tab, colon, vertical bar (or pipe), space.
- Some other characters that may be used as a delimiter are tilde(~), semi-colon
- ASCII includes several control characters that are intended to be used as delimiters. They are: 28 for File Separator, 29 for Group Separator, 30 for Record Separator, and 31 for Unit Separator. But they are not widely used.
See also:- https://en.wikipedia.org/wiki/Delimiter-separated_values
csv standards
https://en.wikipedia.org/wiki/Comma-separated_values mentions three standards
- https://frictionlessdata.io/specs/tabular-data-package/
- http://frictionlessdata.io/specs/data-package/
My thoughts: provides a way to associate schema with the data set. But is somewhat difficult to use.
- https://www.w3.org/TR/tabular-data-model/ - Model for Tabular Data and Metadata on the Web. Too long did not read.
- https://tools.ietf.org/html/rfc4180 - specifies a standard for comma separated values.
My thoughts: No way to describe the schema of data files.