                        README file for MADIS
                          MADIS Version 4.2
                          December 27, 2013
--------------------------------------------------------------------------------

The Meteorological Assimilation Data Ingest System (MADIS) is
dedicated toward making value-added data available from the National
Oceanic and Atmospheric Administration's (NOAA) Earth System Research
Laboratory (ESRL) Global Systems Division (GSD) (formerly the Forecast
Systems Laboratory (FSL)) for the purpose of improving weather
forecasting, by providing support for data assimilation, numerical
weather prediction, and other hydrometeorological applications.


Components
----------

1) A netCDF observational database containing quality controlled meteorological
   observations that can be obtained from MADIS (see below):

   - Surface datasets

     - ASOS METAR reports (NWS & FAA)
     - Non-ASOS METAR reports
     - High frequency METAR reports
     - SAO reports 
     - Buoys, ships, C-MAN stations
     - Modernized NWS Cooperative Observer stations
     - UrbaNet stations
     - Mesonets
     
   - Radiosondes

   - NOAA Profiler Network (NPN) hourly winds

   - Automated aircraft reports (including ACARS, AMDARS, TAMDARS)

   - Aircraft data profiles at airports

   - Hydrological datasets

     - Mesonets

   - Multi-Agency Profilers (MAP)

   - Radiometers

   - Satellite wind datasets

     - GOES satellites, operational winds
     - GOES satellites, experimental winds

   - Satellite soundings datasets

     - POES satellites

   - Satellite radiance datasets

     - POES satellites

   - Snow 

   - Weather In- Situ Deployment Optimization Method (WISDOM) balloon
     winds

2) A Fortran-callable Application Program Interface (API) that allows a user
   to read the database without any knowledge of netCDF.

   The API can also be used to read the point datasets on National Weather
   Service AWIPS systems.

3) Utility programs that can be used to read station information, observations,
   and quality control (QC) information for a single time, and dump these to
   an output text file.  The operation of each program is controlled by a text 
   parameter file that allows the user to exercise all of the options included 
   in the MADIS system.  The programs can be run as needed to access any MADIS 
   data stored on your system, or can be run as a time-scheduled task to get 
   data keyed to the current time.

   The dump programs (written in Fortran) are also meant as tutorials to 
   demonstrate how a user can use the MADIS API calls to write his or her 
   own Fortran programs.

   - sfcdump                    Surface datasets

   - raobdump                   Radiosondes

   - npndump                    NOAA Profiler Network hourly winds

   - acarsdump                  Automated aircraft reports

   - acarspdump                 Aircraft data profiles at airports

   - hydrodump                  Hydrological datasets

   - mapdump                    Multi-Agency Profilers

   - rdmtrdump                  Radiometers

   - satraddump                 Satellite radiance

   - satsnddump                 Satellite soundings

   - satwnddump                 Satellite winds

   - snowdump                   Snow

   - wisdomdump                 WISDOM balloon winds

   In addition, the madis_ingest program is a tutorial to demonstrate how all
   datasets might be read in together for use in an hourly data assimilation
   cycle. 

   The utility programs can be used to access either the FSL database, or the
   AWIPS database.

4) Documentation:

   System
   ------
   INSTALL.unix                 How to install for Unix or Linux
   INSTALL.windows              How to install for Windows
   README                       General package description
   RELEASE_NOTES                Chronology of changes for all versions
   errors.txt                   Error codes, messages, corrective action
   hydro_providers.txt          Hydro data providers available from MADIS
   map_providers.txt            Multi-agency profiler providers available from MADIS
   rdmtr_providers.txt          Radiometer providers available from MADIS
   satrad_providers.txt         Satellite radiance providers available from MADIS
   satsnd_providers.txt         Satellite sounding providers available from MADIS
   satwnd_providers.txt         Satellite wind providers available from MADIS
   snow_providers.txt           Snow providers available from MADIS
   sfc_providers.txt            Surface providers available from MADIS

   Utility Programs
   ----------------
   acarsdump_usage.txt          The *usage.txt files describe how to run each
   acarspdump_usage.txt         dump utility, how to set up the parameter files, 
   hydrodump_usage.txt          etc.
   mapdump_usage.txt            
   npndump_usage.txt
   raobdump_usage.txt
   rdmtrdump_usage.txt
   satraddump_usage.txt
   satsnddump_usage.txt
   satwnddump_usage.txt
   snowdump_usage.txt
   sfcdump_usage.txt
   wisdomdump_usage.txt

   acarsdump.par                Sample parameter files for each dump utility,
   acarspdump.par               these will dump all variables with default
   hydrodump.par                settings on all options.
   mapdump.par                  
   npndump.par                  
   raobdump.par
   rdmtrdump.par
   satraddump.par
   satsnddump.par
   satwnddump.par
   snowdump.par
   sfcdump.par
   wisdomdump.par

   hydrodump1.par               Sample hydrodump.par - specific-provider option.
   mapdump1.par                 Sample mapdump.par - tabular output.
   mapdump2.par                 Sample mapdump.par - specific-provider option.
   npndump1.par                 Sample npndump.par - tabular output.                
   raobdump1.par                Sample raobdump.par - tabular output.
   rdmtrdump1.par               Sample rdmtrdump.par - tabular output.
   rdmtrdump2.par               Sample rdmtrdump.par - specific-provider option.
   satraddump1.par              Sample satraddump.par - specific-provider option.
   satsnddump1.par              Sample satsnddump.par - specific-provider option.
   satwnddump1.par              Sample satwnddump.par - specific-provider option.
   snowdump1.par                Sample snowdump.par - specific-provider option.
   sfcdump1.par                 Sample sfcdump.par file - mesonet-only variables.
   sfcdump2.par                 Sample sfcdump.par file - specific-provider option.



   Variables
   ---------
   acars_variable_list.txt      The *variable_list.txt files list all variables
   acarsp_variable_list.txt     available for each dataset, along with their 
   hydro_variable_list.txt      MADIS variable code names, units, and 
   map_variable_list.txt        explanatory notes.
   npn_variable_list.txt        
   raob_variable_list.txt       
   rdmtr_variable_list.txt
   satrad_variable_list.txt
   satsnd_variable_list.txt
   satwnd_variable_list.txt
   sfc_hfmetar_variable_list.txt
   sfc_sao_variable_list.txt
   sfc_maritime_variable_list.txt
   sfc_metar_variable_list.txt
   sfc_coop_variable_list.txt
   sfc_urbanet_variable_list.txt
   sfc_mesonet_variable_list.txt
   sfc_qc_variable_list.txt
   snow_variable_list.txt
   wisdom_variable_list.txt

   QC Information
   --------------
   qc_info.txt                  Description of the QC info that's available.

   acars_qc_variable_list.txt   The *qc_variable_list.txt files list the QC
   acarsp_qc_variable_list.txt  checks applied to the variables for each 
   hydro_qc_variable_list.txt   dataset.
   map_qc_variable_list.txt     
   npn_qc_variable_list.txt
   raob_qc_variable_list.txt
   rdmtr_qc_variable_list.txt
   satrad_qc_variable_list.txt
   satsnd_qc_variable_list.txt
   satwnd_qc_variable_list.txt
   snow_qc_variable_list.txt
   sfc_qc_variable_list.txt
   wisdom_qc_variable_list.txt

   Programmer Documentation
   ------------------------
   subroutine_list.txt          List of MADIS routines.

   msetdom.txt                  Description of user-callable subroutines that
   msetwin.txt                  aren't demonstrated in the *dump programs.
   time_routines.txt

   For the most current documentation (which can be updated outside of the
   version release cycle), go to:

   http://madis.noaa.gov/madis_api_doc.html

5) Thirteen hours of sample data that can be used to try out the system before
   requesting real-time or archived data from MADIS.  The sample data can be 
   obtained by following the instructions at:

   http://madis.noaa.gov/madis_sample_data.html


Supported Systems
-----------------

The MADIS system has been tested on the following platforms, and
precompiled binary versions of the software are available (see
INSTALL.* for how to obtain the distribution).  Note that the lists of
operating system version numbers and computer models that were tested
represent only those used in testing the current release, and do not
imply that other versions of the operating system or other computer
models in the same family will not work.  Where multiple compilers are
listed for a given platform, the one specified first in the line is
the one used in the precompiled binary distribution.  Starting with
MADIS API version 4.2 only the Linux and Windows versions are tested.
They are tested only 32 byte and 64 byte machines.  Contact 
madis-support@noaa.gov if you have question on another OS.

1) Platform:              Hewlett Packard 
   Operating system:      HP-UX, 10.20
   Models tested:         HP-9000/D270
   Compilers tested:      f77, f90
   Binary portability:    Should work on any HP PA-RISC 1.1 or 2.0 platform
   Last API tested:       3.7

2) Platform:              IBM POWERx, PowerPC 
   Operating system:      AIX, 5.1.0
   Models tested:         E-Server 1600 Cluster
   Compilers tested:      xlf, xlf90
   Binary portability:    Should work on any IBM POWERx or PowerPC platform
   Last API tested:       3.9

3) Platform:              Sun 
   Operating system:      SunOS, 5.10
   Models tested:         Sun Fire V210
   Compilers tested:      f77
   Binary portability:    Should work on any SPARC platform
   Last API tested:       3.9

4) Platform:              Linux
   Operating system:      Red Hat Enterprise Linux Client release 5.10
   Models tested:         Xeon 
   Compilers tested:      g77, pgf90
   Binary portability:    Should work on any Intel or Intel-compatible Linux 
                          platform
   Last API tested:       Current version

5) Platform:              Silicon Graphics 
   Operating system:      IRIX/IRIX64, 6.5
   Models tested:         Origin 200, Indigo 2
   Compilers tested:      f77
   Binary portability:    Should work on any n32 platform
   Last API tested:       3.9

6) Platform:              PC running Windows
   Operating system:      Windows
   Models tested:         Xeon running XP
   Compilers tested:      g77
   Binary portability:    Should work on any Windows XP/Win7 platform
   Last API tested:       Current version


Limitations
-----------

1) If you intend to compile and link either the MADIS API or user-written
   programs that link to the API, Unidata netCDF software must be installed 
   on the system.  For Unix/Linux operating systems, binaries and source code 
   can be obtained from:

   http://www.unidata.ucar.edu/packages/netcdf

   MADIS stopped OPeNDAP/DODS support with version 3.9.  The information 
   below is for historical purpose.

   Software built with the MADIS API can optionally be built with the
   OPeNDAP/DODS netCDF client library in place of the normal netCDF library.
   This would allow you to use the same software to access MADIS netCDF files
   stored on your local disk or served over the web from the MADIS OPeNDAP
   server.  The OPeNDAP/DODS netCDF client library (and other supporting
   software) can be obtained from:

   Current Version       http://www.opendap.org
   Version 3.2.1         ftp://ftp.unidata.ucar.edu/pub/dods/DODS-3.2/3.2.1

   Although the OPeNDAP/DODS interface has only been tested with the Linux
   version of the API, any site that has the OPeNDAP/DODS (version 3.2.1)
   netCDF client library installed should be able to build the MADIS API to 
   link to it by following the instructions in the MADIS makefile.

   For Windows operating systems use the netCDF package that's included in 
   the MADIS binary distribution. 

   In order to support the least-common-denominator RedHat Linux version,
   the MADIS 3.9 binaries are built using RedHat AS 2.1.  The latest version 
   of OPeNDAP/DODS binaries that work with RedHat AS 2.1 is 3.2.1, so that's 
   what's been linked in with the MADIS Linux binaries.  Note that we have 
   been unable to successfully link and/or run with the OPeNDAP/DODS 3.4.3 
   and 3.4.4 versions.  The OPeNDAP/DODS documentation strongly recommends 
   that you should not use their binary versions if you intend to link 
   user-supplied code (e.g., the MADIS API), but you should build 
   OPeNDAP/DODS from source using the same compiler that will be used with 
   your source.  Any MADIS users who want to use the newer OPeNDAP/DODS 
   versions for Linux, or wish to run OPeNDAP/DODS-enabled MADIS API 
   programs on another operating system, should probably take that approach.  
   If you do manage to succeed in this effort, please let us know so we can 
   update this documentation.

2) Fortran logical unit numbers 20-99 are reserved for MADIS library usage.
   (Only a handful are used.  To modify this, change m_getlun.f.)


How to Obtain MADIS Data from ESRL/GSD
--------------------------------------

MADIS subscribers can receive real-time data or obtain access to an
on-line archive (back to July 1, 2001 for most datasets) by requesting
an account on our ftp and/or web server.  For real-time data, LDM
access is also available.

1) Restrictions on distribution

   Some of the datasets are restricted.  There are four distribution
   categories set up to handle the restrictions:

   - ESRL/GSD only - no restrictions
   - NOAA only
   - Government, research, and education organizations
   - Full distribution

   The following datasets are available without restriction:

   - Surface (non-mesonet)
   - Radiosondes
   - NOAA Profiler Network (NPN) hourly winds
   - Hydrological (non-mesonet)
   - Multi-Agency Profilers
   - Radiometers
   - Satellite winds
   - Satellite soundings
   - Satellite radiance
   - Snow
   - WISDOM balloon winds

   The following datasets are available with some restrictions:

   - Automated aircraft reports
   - Aircraft data profiles at airports
   - Surface (mesonet)
   - Hydrological (mesonet)

   The restrictions in the surface and hydrological datasets are on
   specific mesonet providers.  The list of providers currently
   available and their disribution categories can be found at:

   http://madis.noaa.gov/mesonet_providers.html  [surface]
   http://madis.noaa.gov/hydro_providers.html    [hydro]
   
   The aircraft datasets are proprietary to the airlines providing the data.
   Requests for access will be forwarded to them for review.  In general, 
   real-time aircraft reports and profiles are available to organizations 
   conducting research which is likely to benefit the airlines, and to 
   government agencies supporting forecasting operations.  Archived data 
   is available without restriction.

2) Accessing the data (users outside ESRL/GSD)

   To request real-time or archive access to any of the MADIS datasets, 
   please fill out the application form at:

   http://madis.noaa.gov/data_application.html
    
3) Accessing the data (ESRL/GSD users)

   The real-time MADIS data is under /public/data/madis.  ESRL/GSD users of
   the API can point to the files by setting the MADIS_DATA environment 
   variable:

   setenv MADIS_DATA /public/data/madis

   For archive requests, put in an OTRS request for the desired dates and 
   directories.


Questions/Comments
------------------

Please send your questions/comments to:

        madis-support@noaa.gov

