data.export

data.export is used to reformat CPD2 style data into formats more easily understood by other programs. It takes its input from standard input and produces output on standard output based on various switches set on the command line.

data.export [--mode=excel|xl|csv|idl|r|R|archive|idllegacy|cpd2] 
            [--merge=none|last|interpolate]
            [--join=csv|space] 
            [--station=on|off] 
            [--date-epoch=on|off] [--date-excel=on|off] [--date-iso=on|off] 
            [--date-fyear=on|off] [--date-julian=on|off]
            [--date-yeardoy=on|off] [--date-doy=on|off]
            [--mvc-type=blank|mvc|na] [--mvc-flag=follow|end|off]
            [--flags-type=breakdown|0x|default] [--header-mark=...]
            [--header-names=on|off|-1] [--header-desc=on|off]
            [--header-namesstderr=on|off] [--header-wavelength=on|off]
            [--header-cpd2global=on|off] [--header-mvcs=on|off]
            [--format-default=off|<format>] [--format-<name>=<format>]

The mode argument is processed before any other arguments, so further arguments can be used to override the defaults set by the mode.

Select mode. Setting a mode sets the defaults for all parameters relevant to that mode.

Export data in CSV format with year and DOY timestamps.

Normal usage:

... | data.export --mode=csv > data.csv

Export data in CSV format for importing into Excel.

Normal usage:

... | data.export --mode=excel > data.csv

Export data in space separated and converted format for reading with IDL. Also generates a header on standard error with the names and index (zero-based) of each column.

Normal usage:

... | data.export --mode=idl > data

Will generate a data file and a header on standard error with the column names as above.

Export data in CSV format for importing into R.

Archive export mode.

IDL legacy export mode.

Reformatted CPD2 style data. This mode does not accept any other format switches.

Select merging mode for data inputs. Note that this has no effect when operating on already merged data (such as data generated from data.avg). The default is none.

• none - Disable merging. Values not present at any time are set to MVC.
• last - Use the most recent value for any variables missing at a given time.
• interpolate - Interpolate values over missing times for variables.

Select joining mode. Joining determines how values are combined into record lines and how they are separated within those lines.

Comma separated value join mode. If values contain spaces or any other condition that would require quoting, that value is quoted as per Text::CSV_XS.

Space separated values. No handling for values with embedded spaces; do not use on string data.

Enable or disable the leading column with the station code.

Enable or disable outputting epoch time (seconds since Jan 1 1970).

Enable or disable outputting Excel formatted time (yyyy-mm-dd hh:mm:ss).

Enable or disable outputting ISO formatted time (yyyy-mm-ddThh:mm:ssZ).

Enable or disable outputting fractional year.

Enable or disable outputting fractional Julian day.

Enable or disable outputting a year and fractional day of year.

Enable or disable outputting a fractional day of year.

Select format for undefined values.

• blank - Leave that position as a zero length blank.
• mvc - Use the missing value code for that column.
• na - Print the string 'NA' for all missing values.

Set the mode for MVC flags for each column. The flag is either 0 (valid value) or 1 (invalid).

• follow - The column with the MVC flag immediatly follows the value column.
• end - All MVC flags are placed at the end of the value columns in the same order as the value columns.
• off - Disable MVC flags.

Enable breakdowns of bitwise flags. When enabled instead of a single column with the bit set of flags N columns of values of either 0 or 1 will be generated for each bitwise flag. N is the number of bits that the bit set can contain (e.g. 16 for a four digit hexadecimal number).

Insert '0x' before any hexadecimal numbers.

Do not reformat flags.

Set the header marker. This string is prepended to all header lines.

Enable or disable outputting a header line before the data begins with the names of each variable. The names do not contain spaces. If set to -1 then the names are output immediately before the actual data instead of at the start of the header lines.

Enable or disable outputting a header line with the field description of each variable before data. The field description may contain spaces.

Enable or disable printing a header line to standard error with the names and column numbers of the data. The line is comma separated values of the form “NAME:INDEX”. Where the name is the name as –header-names above and the index is the zero-based index of that column.

Enable or disable printing a header line with the wavelength information for each variable.

Enable or disable printing header lines for all CPD2 global headers.

Enable or disable printing a header line with the MVC for each variable.

Set the default format for all value columns or disable default formatting. When a default format is set, all columns use that format unless overridden with –format-<name> (see below). The format is a CPD2 extended printf code.

Override the format for a given variable (specified by <name>). The format is a CPD2 extended printf code.

Example:

1. -format-BaG_A11=%07.2f

data.get sgp S11a 2008:10 2008:11 | data.export --mode=excel > data.csv

data.avg sgp S11a 2008:10 2008:11 | data.export --mode=idl --flags-type=default

data.get sgp S11a 2008:10 2008:11 | data.export --mode=excel --station --date-fyear > data.csv
  

 data.get app S11a,A11a,N71a,X1a,X2a,XIe 2010:1 1d avgh | data.export --mode=archive > aerosol_main_app_2010_hr.csv