---------------------------
MADIS Sfcdump Program Usage
---------------------------


1.0  Introduction
-----------------

The MADIS sfcdump program can be used to read surface station information, 
observations, and quality control (QC) information for a single time, and dump 
this to an output text file.  The operation of the program is controlled by a 
text parameter file that allows the user to exercise all of the options included 
in the MADIS system for surface data.  The program can be run as needed to 
access any MADIS data stored on your system, or can be run as a time-scheduled 
task to get data keyed to the current time.

This document describes how to setup the parameter file, how to run the 
program, and what to expect in the way of outputs.  Fortran programmers who 
wish to write their own programs to read MADIS data can find much of the same 
information in the sfcdump.f source code in the src directory.  In addition to 
being used for the dump utility, the code is annotated to provide a tutorial for 
programmers.


2.0  How to Run
---------------

1.  Make sure you have the necessary environment variables defined:

    - MADIS_STATIC  points to the MADIS static files directory

    - MADIS_DATA    points to the top-level MADIS data directory

    AWIPS users should point MADIS_DATA to FXA_DATA.

    See the INSTALL.* files for information on how to set environment 
    variables on your system.

2.  Setup the desired parameters in a sfcdump parameter file.

3.  Move to the directory in which you'll be running the program.

4.  Copy the parameter file to the run directory, and make sure that its 
    name is "sfcdump.par".

5.  Run the program.  (bin/sfcdump.exe on Unix/Linux, bin/sfcdump on Windows)

6.  Outputs:

    a.  An activity log showing the selections made from the parameter file,
        as well as any errors that resulted will be written to standard output.
    
    b.  The data that's been obtained will be written to sfcdump.txt in the
        run directory.


3.0  How to Setup the Parameter File
------------------------------------

These sample parameter files can be found in the doc directory:

- sfcdump.par    Dumps all variables, using defaults for all options.

- sfcdump1.par   Dumps all mesonet variables, for mesonet providers only, using 
                 defaults for all options.

- sfcdump2.par   Example of how to specify that data be dumped for only certain
                 data providers, and for only a few variables.

The parameter file is split into several sections, with each section delimited
by a line of "----------"'s.  With a couple of exceptions noted below, you
must include all lines that are within each section (in other words, the 
sections are "fixed length").  Each of the sections is described below, in the 
order in which they appear in the parameter file.


3.1  Database Section
---------------------

---------------------------------------------------------------------------------
              Database                                    FIXED LENGTH SECTION
              --------
FSL           'FSL' or 'AWIPS'
---------------------------------------------------------------------------------

If you have obtained MADIS files from FSL, leave the line as shown above.  If
you're reading AWIPS data, enter AWIPS at the beginning of the line.


3.2  Time Window Section
------------------------

---------------------------------------------------------------------------------
              Time Window (use 0,0,0 for default)         FIXED LENGTH SECTION
              -----------
-30           Number of minutes relative to nominal time at which to start window
30            Number of minutes relative to nominal time at which to end window
3             0 - return all records within the file containing nominal time
              1 - return one record per fixed station, closest to nominal time
              2 - return one record per fixed station, closest to start of window
              3 - return one record per fixed station, closest to end of window
              4 - return all records within *window*
---------------------------------------------------------------------------------

The user selects the nominal time of the data to be returned in the Time
Selection section below.  Time "windowing" is used so that the program will
know how close in time each station's observation time needs to be to the 
nominal time in order to be included, and whether or not duplicate records from 
the same station that are within the time window should be included.  The time 
window is specified by selecting the number of minutes before and after the 
nominal time to be used for the window, and how duplicates should be handled.  

If the default values (0,0,0) are selected, then all records (including multiple 
reports from the same station) will be returned from the file containing the 
nominal time.  For non-mesonet stations, this will return observation times 
between 15 minutes before the hour and 44 minutes after the hour.  For mesonet 
stations, data will cover from the hour until 59 minutes after the hour.

Surface stations report observation times around the hour, and many stations 
report multiple times per hour, so different time window specifications can 
produce very different sets of data.

The settings shown in the sample section above (-30,30,3) will return only 
one record per station within the specified time window (-30 minutes --> 
nominal time --> +30 minutes).  If a station has multiple reports within the 
time window, only the latest one within the window will be returned.  For 
example, if the nominal time is 12:15 and a station has observations available 
at 11:51, 12:01, 12:11, and 12:21, the 12:11 observation will be returned.

If you are running the program as a scheduled task with the nominal time
keyed to the current time, one recommended window setting would be -60,0,3.
This would return the most current observation for every station that had
reported within the last hour.

The maximum window is 180 minutes.  Also note that the window is inclusive,
that is, using start & end minutes -30,30 with a nominal time of 12:00 will
return observations with times ranging from 11:30 through 12:30.


3.3  Provider Section
---------------------

---------------------------------------------------------------------------------
              Providers (see doc/sfc_providers.txt)       VARIABLE LENGTH SECTION
              ---------
0             0 - all providers 
              N - list of N provider names follow this line 
---------------------------------------------------------------------------------

Surface datasets include many different networks run by different "providers".  
If desired, the user can select only a subset of the total by specifying which 
providers are to be included or excluded.

The default (shown above) is to enable data from all surface providers.

See sfcdump2.par for an example of how to edit this variable-length section
to specify a list of providers to be enabled.

Here's the list of provider names recognized by MADIS:

  'ALL-SFC'   All surface providers
  'ALL-MESO'  All mesonet providers
  'ALL-MTR'   All METAR reports
  'ASOS'      ASOS METAR reports (NWS & FAA)
  'OTHER-MTR' Non-ASOS METAR reports
  'ALL-HFMTR' All high frequency METAR reports
  'ASOS-HFM'  High frequency ASOS METAR reports
  'OTHER-HFM' High frequency non-ASOS METAR reports
  'SAO'       SAO reports 
  'MARITIME'  Buoys, ships, C-MAN stations
  'COOP'      Modernized NWS Cooperative Observer stations
  'UrbaNet'   UrbaNet
  'NEPP'      New England Pilot Project stations
  'HCN'       Historical Climate Network Modernization (HCN-M)
  'CRN'       Climate Reference Network

For a list of the provider names for mesonets that are currently available from 
FSL, see:

  http://madis.noaa.gov/mesonet_providers.html

For AWIPS users, the "Data provider" specified for each mesonet in the 
/data/fxa/LDAD/data/*.desc files is what's used for the MADIS provider name.


3.4  Domain Filter Section
--------------------------

---------------------------------------------------------------------------------
              Domain Filter                               FIXED LENGTH SECTION
              -------------
0             0 - don't filter
              1 - return stations within latitude/longitude corners
              2 - return stations within specified Polar Stereographic grid
              3 - return stations within specified Lambert Conformal Conic grid

              Latitude/Longitude Corners (lines skipped if not Domain Filter 1)
              --------------------------
37.0          SW corner latitude (north)   
-109.0        SW corner longitude (east)
41.0          NE corner latitude (north)
-102.0        NE corner longitude (east)

              Polar Stereo Specification (lines skipped if not Domain Filter 2)
              --------------------------
190500.0      Grid box size (meters)
7.838         Latitude (north) of 1st grid point (lower left = SW)
-141.028      Longitude (east) of 1st grid point (lower left = SW)
-95.0         Orientation longitude (east)                 
33.0          I-coordinate of pole
45.0          J-coordinate of pole
60.0          Lat (north) at which X-Y scale is true
65            Number of grid points in X-direction (I dimension)
43            Number of grid points in Y-direction (J dimension)

              Lambert Conformal Conic Specification (lines skipped if not Domain Filter 3)
              -------------------------------------
40635.25      Grid box size (meters)
12.19         Latitude (north) of 1st grid point (lower left = SW)
-133.459      Longitude (east) of 1st grid point (lower left = SW)
-95.0         Orientation longitude (east)
25.0          Latitude (north) of first Lambert Conformal tangent
25.0          Latitude (north) of second Lambert Conformal tangent
185           Number of grid points in X-direction (I dimension)
129           Number of grid points in Y-direction (J dimension)
---------------------------------------------------------------------------------

The horizontal domain can be limited to match one of two map projections (polar 
stereographic, Lambert conformal conic) or it can be limited to a 
latitude/longitude rectangle.  Only stations within the domain will be returned.  
If one of the map projections is selected, any winds (of any form) returned will
be rotated to match the projection.  

If the default is selected (as above), no domain filtering will be performed,
and all available stations will be returned, regardless of their location.

(The example Latitude/Longitude specification shown above is for the state of
Colorado, the example Polar Stereographic specification defines AWIPS grid 202 
(NATIONAL-CONUS), and the example Lambert Conformal specification defines AWIPS 
grid 212 (REGIONAL-CONUS-DOUBLE RESOLUTION).)
 
If one of the map projections has been selected for the horizontal domain, the 
station latitude and longitude will be converted to a grid index into the 
2-dimensional grid (RI,RJ).  Note that this works for grids laid out this way:

                     NE (I=NX,J=NY)

  SW (I=1,J=1)

The calculated index will appear in sfcdump.txt in the "Grid I" and "Grid J"
columns.  If a map projection isn't being used, Grid I and Grid J will be
output as zeros.


3.5  QC Filter Section
----------------------

---------------------------------------------------------------------------------
              QC Filter                                   FIXED LENGTH SECTION    
              ---------
0              0 - none
               1 - coarse
               2 - screened
               3 - verified
              99 - highest possible
---------------------------------------------------------------------------------

The primary variables in all MADIS datasets have been screened by automated 
quality control (QC) checks.  Different variables have different levels of QC, 
or "stages", applied to them.  For example, temperature may have level 1 checks 
(validity), level 2 checks (internal consistency, temporal consistency), and 
level 3 checks (spatial consistency).  Note that each level of QC incorporates 
the previous levels, i.e., an ob that has passed level 3 has also passed any 
applicable level 1 and 2 checks.  See sfc_qc_variable_list.txt for more 
information.  (Level 1 checks = "coarse", level 2 = "screened", level 3 =
"verified".)

Each variable that gets dumped will also have its associated QC information
written to sfcdump.txt.  This includes a QC data descriptor (a single 
character summarizing the info), and bitmasks showing which QC checks were
applied and the results of the checks.  These appear in the "QCD", "QCA", and
"QCR" columns of the output.  For details on how to interpret these items see
qc_info.txt.

If the user doesn't want to interpret the QC info, he or she can limit the 
returned obs to selectable levels of quality.  In that case, obs that don't 
meet the selected QC level will be omitted.

There are three ways that the user can request and process the QC information:

  1) Use the default setting (shown above) which will disable QC filtering.
     Then you can examine the QC info before deciding whether or not to use 
     each observation.  This allows for the maximum amount of control over 
     what's available.  

  2) If you know ahead of time that the same level of QC is available for all 
     variables you'll be using, select that level.

  3) To require that all variables pass the highest level of QC that are 
     specified for them, select QC level 99.

User's of MADIS data who want the data filtered by QC results, but don't want to
interpret the QC information themselves, should select QC level 99.


3.6  Time Selection Section
---------------------------

---------------------------------------------------------------------------------
	      Time Selection                              FIXED LENGTH SECTION
              --------------
1             0 - Julian format (YYJJJHHMM)
              1 - Month/Day format (YYYYMMDD_HHMM)
0             0 - Use the nominal time line below
              1 - Use the current time as the nominal time
20040824_1200 Nominal time (in selected format)
---------------------------------------------------------------------------------

All MADIS data are time-stamped in Universal Coordinated Time (also called GMT).
Two formats are supported, both for inputting the nominal time in the 
parameter file, and for outputting the time in the sfcdump.txt output file.

"Julian format" is in the form of "YYJJJHHMM", where:

  YY  = 2-digit year (good from 1980 - 2179)
  JJJ = Julian date of the year (001-366)
  HH  = Hour (00-23)
  MM  = Minute (00-59)

"Month/Day" format uses "YYYYMMDD_HHMM", where:

  YYYY = 4-digit year (still only good from 1980 - 2179)
  MMDD = Month/Day 
  HHMM = Hour/Minute 

The nominal time shown above in the Month/Day format translates to 042371200
in the Julian format.

Nominal times can be specified at any minute, not just at the top of the hour 
(minute 00).

To run the program as a scheduled task, specify that the current time be
used as the nominal time and set an appropriate time window.


3.7  Station Selection Section
------------------------------

---------------------------------------------------------------------------------
	      Station Selection                           FIXED LENGTH SECTION
              -----------------
0             0 - Get all stations
              1 - Get only the station whose name is on the next line
KADC          Single station name
---------------------------------------------------------------------------------

The default shown here is to get data from all stations that meet the criteria
already specified above.  If desired, you can specify a single station to be
dumped.


3.8  Output Options Section
---------------------------

---------------------------------------------------------------------------------
	      Output Options                              FIXED LENGTH SECTION
              --------------
0             0 - Lat/lon with 2 digits after decimal place 
              1 - Lat with 5 digits after decimal place, lon with 4 digits

0             0 - Text output with headers (original format)
              1 - XML output format  
              2 - Comma-separated-variable (CSV) text output without QC info
              3 - CSV text output with QC data descriptors
              4 - CSV text output with full QC information
              5 - Extended CSV text output with QC data descriptors - format D
              6 - Extended CSV text output with full QC information - format F
              7 - Extended CSV text output with QC data descriptors - format D+DS+R
              8 - Extended CSV text output with full QC information - format F+DS+R

0             0 - When using CSV, use -99999.000000 for missing fields
              1 - When using CSV, use blanks for missing fields (Excel-friendly)
---------------------------------------------------------------------------------

The latitude and longitude can either be output using two digits after
the decimal point (the original setting prior to adding this option to
sfcdump) or the full 7-digits of possible precision can be selected.
This means latitude will have 5 digits after the decimal point and
longitude 4 digits.  Note, however, that we only have 2-digit precision
for ASOS, OTHER-MTR, SAO, and most MARITIME stations.  The amount of
precision available for the mesonets varies by network.

The output can be either text or XML.  The text format with headers is
one line per variable/station/time record.  The CSV text format is one 
line per single station and time for all variables.  The maximum length 
of the line is 560 characters.  This is normally at least 14 variables 
(and optionally, their QC information).  The large character variables 
like AUTORMK, OPERRMK, STALOC, and SKYCOV can reduce the allowed number 
of variables.  The "Extended CSV text" for 5 and 6 outputs a header line 
for the first line of the data.  The  "Extended CSV text" for 7 and 8 
options are for MADIS distribution server use.


For details about the output formats see:

http://madis-data.noaa.gov/sfcdump_help.html#Output_Selection


3.9  Variables Section
----------------------

---------------------------------------------------------------------------------
              Variables (1 per line, until end of file)   VARIABLE LENGTH SECTION
              --------- (see doc/sfc_variable_list.txt)
TD     
RH     
Q
DPD
---------------------------------------------------------------------------------

This is the last section in the parameter file.  The code names for the variables
to be dumped are listed, one per line, until the end of file is reached.  (The
section delimiter line shown here is simply for readability.  If you have this 
line or any other non-variable line in the real parameter file you'll get an 
error.)

The list of code names is found in the first column of sfc_variable_list.txt,
along with the units for each variable, the maximum QC level, and many other
details.

In the above example, note that we're dumping the four moisture variables
supported by MADIS (dewpoint temperature, relative humidity, specific humidity, 
and dew point depression).  For many variables, you can select multiple forms
and MADIS will do the calculations for you.  For full information on what
variables are available and how to interpret them, see sfc_variable_list.txt.
