CPD3 Frequently Asked Questions
Organization and General Information
What are the sections of this FAQ
- Acquisition System (real-time operation)
Where can I find additional information
See http://www.esrl.noaa.gov/gmd/aero/about/faq.html for general aerosol system questions.
See CPD2 FAQ for questions about the CPD2 system
See CPD2 records and CPD2 variables for information about record and variable names and definitions in the CPD2 system. CPD3 uses a different internal approach to representing data, although many of the CPD2 names of records and variables can still be used. See CPD3 data structures for variables used in CPD3.
See CPD3 programs and database for a list of programs available in CPD3 database system.
Detailed help on any CPD3 program is available by supplying the –help argument, and detailed help on any argument is available by using the –help=argname argument, for example
da.export --help da.export --help=count
In this document “$STATION” refers the the lower case station code (e.x. “mlo”) of the station in question.
Commonly-used Programs
General data access
- da.get - To access data in native CPD3 format.
- da.avg - To generate averages.
- da.export - To reformat data into other styles (e.x. CSV).
- da.select - Selects data to get, with fine control over details
- da.where - Selects data based on conditional tests
- cpd3messagelog - Get entries from on-line message log
- da.generate.edited - Generates edited data for a station, allowing use of different editing profiles to sprcify different sets of data or methods of generating edited data.
Editing data
- da.pass - To pass data into the clean archive.
Viewing data
Getting Information about Data and Instruments
How do I get a list of all the variables available from a station
da.summary.general spo everything forever raw_meta da.summary.general --description spo everything 2016 raw_meta da.summary.general --description spo '.*_A82' 2016 raw_meta
How do I get a timeline of instrument availability
da.summary.timeline --interval=1mo --timeline=vertical bld everything 2015 now raw_meta da.summary.timeline --interval=1y mlo everything forever raw_meta
How do I get general information on data availability
da.summary.general --mode=variable --description spo '.*_A81' 2014
How do I calculate summary statistics for a specified time interval (e.g., overnight nephelometer zero check)
da.summary.general --rows=flavorless thd 'B.*_S11' 51.1 51.72
How do I examine the user entries in the on-line message log
cpd3messagelog bnd 1w now
Editing Data
How do I bring up the Mentor Edits window without loading a lot of data
The easiest way is to load the averaged data, which goes quickly.
cpx3 bnd 2016-05-01 1mo avg
Alternatively, you can use one of the alternative data profiles that only includes a few variables, or that may not include any variables for your station.
cpx3 --profile=met bnd 2016-05-01 1mo
If you use the latter approach, you need to click on the “Display” button in the Edit Directives window and select “Display all profiles”.
How do I interpolate across a gap in the data
This is done with a script edit, triggered by a flag that is associated with the data gap.
The code below implements an interpolation edit. It is entered in the “Script” window for a mentor edit of type “Script Code General Handler”. The variable to be edited is selected in the “Input” window (BeB_E11 for the example here).
var valueBeforeGap = undefined;
var timeBeforeGap = undefined;
var inGap = false;
include('Interpolation');
data.install_value(function(point) {
if (inGap) {
if (ndefined(point.value)) {
var interpolator = new CPD3.Interpolation.Linear(
[timeBeforeGap, point.start],
[valueBeforeGap, point.value]);
for (var i=0; i<data.length; i++) {
var modify = data[i];
if (ndefined(modify.value))
continue;
modify.value = interpolator.value(modify.start);
}
inGap = false;
data.releaseAll();
data.autorelease = 0;
}
} else {
if (ndefined(point.value)) {
valueBeforeGap = point.value;
timeBeforeGap = point.start;
} else {
inGap = true;
data.autorelease = undefined;
}
}
})
Next, the trigger conditions for activating the script need to be specified. For the example here, the blue channel from a CAPS extinction monitor (E11) is to be interpolated across gaps when the flags “Zero” or “Blank” are set. Because there are two triggers, the trigger is of type “Logical OR”. A symmetric extension of 1 minute is specified to include the valid data points at the start and end of the data gap. The “Add” button is clicked twice to add the two logical conditions that comprise the trigger, and the type of each logical condition is “Flags”. The flag variable for both logical tests is F1_E11 for the example here. Clicking on the “Flags” button allows the flag name to be entered (“Zero” for the first logical test, and “Blank” for the second logical test). The resulting screen for the trigger edit is shown below.
The raw data (with gaps) and edited data (interpolated across gaps) are shown below for three CAPS instruments, which required separate edits for each wavelength.
Working with Data
How do I exclude the summary statistics from averaged data
This is done in da.export, not da.avg
... | da.avg ... | da.export —stddev=0 —count=0
How do I calculate the average of all the data, i.e., not broken up into blocks
... | da.avg —interval=undef | ...
However, special handling is needed if there are breaks in the data, e.g., nephelometer zeros or cut-size changes, because CPD3 was designed to “do the right thing” by default, so it ends up being complicated when you want to override the intelligence. Normally, da.avg will break the data into hourly averages because the metadata for the zeros specifies that they shouldn't be averaged (normally correct, you don't want zeros being folded into hourly averages if you put the two together, since the zero specify “segments” rather than values). Use da.select to handle this special case:
... | da.select --path="^Smoothing" --path-apply=:raw_meta: --explicit-meta=true --path-remove=true | da.avg --interval=undef | ...
How do I select data for a particular time period from an input file
da.select --include=.*,2012:1,2012:2 inputfile.cpd3
How do I select a particular variable from a data stream
... | da.select --include=X1c_A81 | ...
How do I remove the size-cut distinction from variables in a data stream
... | da.select --include=':::cover:!' --set-flavor-remove=cover | ...
–alternatively–
... |da.select --set-flavor-remove=pm1 | da.select --set-flavor-remove=pm10 | ...
How do I exclude nephelometer data when in zero or blanking mode
... |da.where --exclude='F1_S11.hasFlag("Zero") || F1_S11.hasFlag("Blank")' | ...
How do I include only nephelometer data where RH>40%
... | da.where --include='U_S11 > 40' | ...
How do I calculate absorption coefficients with the CTS correction
da.generate.edited --profile=cts bnd 2015-01-01 2015-01-02 | da.select --capture-variable='(.*)_A1(.)' --set-apply='.*_A(.+)' --set-variable='${1}_A9${2}'
This will create new virtual instruments A9x representing the CTS-corrected data from all A1x instruments.
It is possible to merge the virtual instruments back into the CPD3 data stream for combined analysis, e.g., compare the Bond and CTS corrections.
da.mux \
"|da.get bnd '.*_A11' 2015 clean" \
"|da.generate.edited --profile=cts bnd 2015 \
|da.select --exclude=:avg: --capture-variable='(.*)_A1(.)' --set-apply='.*_A(.+)' --set-variable='${1}_A9${2}'" \
> mergeddata.cpd3
da.show.density --x=BaG_A11 --y=BaG_A91 mergedata.cpd3
For efficiency in doing further analysis, it is recommended to save the results of the da.mux program to a working file.
How do I merge data from multiple CPD3-format files
da.mux file1.cpd3 file2.cpd3 file3.cpd3 | ...
Plotting and Analyzing Data
Project-specific Questions
What is the percentage of hours that the nephelometer RH was above 40% at EGB and is there any any seasonal dependence
data.get egb S11a 2010 2014 avgh | da.select --include=U_S11 | da.where --include='U_S11 > 40' | da.avg --interval=1mo | da.select --include=':::cover:!' --set-flavor-remove=cover | da.export
Change the –interval=1mo to –interval=undef to get a single coverage number for the whole time.
Is there any annual cycle in the % of hours for which Bsp(PM1) < 1.0 Mm-1 at EGB
data.get egb S11a 2010 2014 avgh | da.select --include=::BsG_S11:pm1 | da.where --include='BsG_S11 < 1' | da.avg --interval=1mo | da.select --include=':::cover:!' --set-flavor-remove=cover | da.export
How do I calculate run averages from a lab experiment
Use the “–mean” switch to da.export to have it output unweighted means from the statistics calculation when available. When it's not available (no averaging done) it will output the conventional values instead.
da.generate.edited --profile=aerosol_native `query.run 2` |da.select --include="Ba[BGR]_A1[145]" |da.avg --interval=undef |da.export --mean |transpose
This bash script provides an alternative approach.
#!/bin/bash
RUN=7
TIMES=`/aer/sfa/work/GMD_lab_absorption_study_2015/runs/query ${RUN}`
echo '<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE cpd3data>
<cpd3data version="1.0">
<value station="sfa" archive="segment" variable="run" flavors=""
priority="0" type=""
start="'${TIMES/%\ */}'" end="'${TIMES##* }'">
</value>
</cpd3data>' |
da.multiplex "| da.generate.edited sfa ${TIMES}" - |
da.smooth.segment --segment=':segment:run' |
da.select --include=':clean:BsG_S11' |
da.export
Acquisition System
Configuration
How do I view the local configuration?
da.config.text bnd acquisition 1d now configuration