Data analysis

From raju

gathering data

  • If you are having trouble dumping data over long horizons, dump data over small horizons and repeat. For example, instead of dumping trade data for each month, dump daily trade data and repeat for each day.

Visualizing data

  • For three variables - plot correlation between two variables for different quintiles of the third variable.

Ex:- plot correlation between the Barclays Capital US High Yield Index and the S&P 500 Index for different VIX Quintiles.

delimited separated values

  • Some common characters used to separate the values are comma, tab, colon, vertical bar (or pipe), space.
  • Some other characters that may be used as a delimiter are tilde(~), semi-colon
  • ASCII includes several control characters that are intended to be used as delimiters. They are: 28 for File Separator, 29 for Group Separator, 30 for Record Separator, and 31 for Unit Separator. But they are not widely used.

See also:- https://en.wikipedia.org/wiki/Delimiter-separated_values

csv standards

https://en.wikipedia.org/wiki/Comma-separated_values mentions three standards

My thoughts: provides a way to associate schema with the data set. But is somewhat difficult to use.

My thoughts: No way to describe the schema of data files.