NASA Earth Data

Written by Minh Phan

This tutorial serves to provide one of many ways a user can download data from NASA’s EarthData database, mostly with datasets (collections in EarthData’s terminology) hosted in the cloud.

Authorize credentials

To download data from NASA’s Earth Data database, it’s recommended that you set up a .netrc credential file so that you don’t have to manually log in every time you run a downloading script. To do this, consult 2021 Cloud Hackathon’s tutorial here. Make sure to register an account with Earth Data first before following the tutorial.

When you finished implementing the .netrc file, continue with the tutorial below

Import necessary libraries

import xarray as xr
import earthaccess
import numpy as np
import pandas as pd
import os, glob

Download data to your local machine using earthaccess library

Earthaccess library streamlines your downloading, slicing, and searching for granules easier than ever. For cloud-hosted datasets (which is what this tutorial best works with), we choose to download granules to the local machine instead of streaming them to the working Python scripts as some users may not be physically available in the us-west region for streaming to be effective. Local downloading may result in heavy file sizes, but is consistent, and we am also providing some tweaks to save as much as you can, especially if your research interest area requires a long temporal range and does not cover globally.

# Log in using .netrc file
auth = earthaccess.login(strategy="netrc")

We are already authenticated with NASA EDL

Streaming granules

# EarthAccess's approach to collecting granules
results = earthaccess.search_data(
    short_name='MUR-JPL-L4-GLOB-v4.1',
    cloud_hosted=True,
    bounding_box = (60, 5, 80, 25),
    temporal=("2003-01-01", "2003-02-28")
)

Granules found: 59

files = earthaccess.open(results) # s3

 Opening 59 granules, approx size: 0.0 GB

ds = xr.open_mfdataset(files)

ds

<xarray.Dataset>
Dimensions:           (time: 59, lat: 17999, lon: 36000)
Coordinates:
  * time              (time) datetime64[ns] 2003-01-01T09:00:00 ... 2003-02-2...
  * lat               (lat) float32 -89.99 -89.98 -89.97 ... 89.97 89.98 89.99
  * lon               (lon) float32 -180.0 -180.0 -180.0 ... 180.0 180.0 180.0
Data variables:
    analysed_sst      (time, lat, lon) float32 dask.array<chunksize=(1, 17999, 36000), meta=np.ndarray>
    analysis_error    (time, lat, lon) float32 dask.array<chunksize=(1, 17999, 36000), meta=np.ndarray>
    mask              (time, lat, lon) float32 dask.array<chunksize=(1, 17999, 36000), meta=np.ndarray>
    sea_ice_fraction  (time, lat, lon) float32 dask.array<chunksize=(1, 17999, 36000), meta=np.ndarray>
Attributes: (12/47)
    Conventions:                CF-1.5
    title:                      Daily MUR SST, Final product
    summary:                    A merged, multi-sensor L4 Foundation SST anal...
    references:                 http://podaac.jpl.nasa.gov/Multi-scale_Ultra-...
    institution:                Jet Propulsion Laboratory
    history:                    created at nominal 4-day latency; replaced nr...
    ...                         ...
    project:                    NASA Making Earth Science Data Records for Us...
    publisher_name:             GHRSST Project Office
    publisher_url:              http://www.ghrsst.org
    publisher_email:            ghrsst-po@nceo.ac.uk
    processing_level:           L4
    cdm_data_type:              grid

Indexes: (3)

time

PandasIndex

PandasIndex(DatetimeIndex(['2003-01-01 09:00:00', '2003-01-02 09:00:00',
               '2003-01-03 09:00:00', '2003-01-04 09:00:00',
               '2003-01-05 09:00:00', '2003-01-06 09:00:00',
               '2003-01-07 09:00:00', '2003-01-08 09:00:00',
               '2003-01-09 09:00:00', '2003-01-10 09:00:00',
               '2003-01-11 09:00:00', '2003-01-12 09:00:00',
               '2003-01-13 09:00:00', '2003-01-14 09:00:00',
               '2003-01-15 09:00:00', '2003-01-16 09:00:00',
               '2003-01-17 09:00:00', '2003-01-18 09:00:00',
               '2003-01-19 09:00:00', '2003-01-20 09:00:00',
               '2003-01-21 09:00:00', '2003-01-22 09:00:00',
               '2003-01-23 09:00:00', '2003-01-24 09:00:00',
               '2003-01-25 09:00:00', '2003-01-26 09:00:00',
               '2003-01-27 09:00:00', '2003-01-28 09:00:00',
               '2003-01-29 09:00:00', '2003-01-30 09:00:00',
               '2003-01-31 09:00:00', '2003-02-01 09:00:00',
               '2003-02-02 09:00:00', '2003-02-03 09:00:00',
               '2003-02-04 09:00:00', '2003-02-05 09:00:00',
               '2003-02-06 09:00:00', '2003-02-07 09:00:00',
               '2003-02-08 09:00:00', '2003-02-09 09:00:00',
               '2003-02-10 09:00:00', '2003-02-11 09:00:00',
               '2003-02-12 09:00:00', '2003-02-13 09:00:00',
               '2003-02-14 09:00:00', '2003-02-15 09:00:00',
               '2003-02-16 09:00:00', '2003-02-17 09:00:00',
               '2003-02-18 09:00:00', '2003-02-19 09:00:00',
               '2003-02-20 09:00:00', '2003-02-21 09:00:00',
               '2003-02-22 09:00:00', '2003-02-23 09:00:00',
               '2003-02-24 09:00:00', '2003-02-25 09:00:00',
               '2003-02-26 09:00:00', '2003-02-27 09:00:00',
               '2003-02-28 09:00:00'],
              dtype='datetime64[ns]', name='time', freq=None))

lat

PandasIndex

PandasIndex(Index([-89.98999786376953,  -89.9800033569336, -89.97000122070312,
       -89.95999908447266, -89.94999694824219, -89.94000244140625,
       -89.93000030517578, -89.91999816894531, -89.91000366210938,
        -89.9000015258789,
       ...
         89.9000015258789,  89.91000366210938,  89.91999816894531,
        89.93000030517578,  89.94000244140625,  89.94999694824219,
        89.95999908447266,  89.97000122070312,   89.9800033569336,
        89.98999786376953],
      dtype='float32', name='lat', length=17999))

lon

PandasIndex

PandasIndex(Index([-179.99000549316406, -179.97999572753906, -179.97000122070312,
        -179.9600067138672,  -179.9499969482422, -179.94000244140625,
       -179.92999267578125,  -179.9199981689453, -179.91000366210938,
       -179.89999389648438,
       ...
        179.91000366210938,   179.9199981689453,  179.92999267578125,
        179.94000244140625,   179.9499969482422,   179.9600067138672,
        179.97000122070312,  179.97999572753906,  179.99000549316406,
                     180.0],
      dtype='float32', name='lon', length=36000))

Attributes: (47)

Conventions :: CF-1.5
title :: Daily MUR SST, Final product
summary :: A merged, multi-sensor L4 Foundation SST analysis product from JPL.
references :: http://podaac.jpl.nasa.gov/Multi-scale_Ultra-high_Resolution_MUR-SST
institution :: Jet Propulsion Laboratory
history :: created at nominal 4-day latency; replaced nrt (1-day latency) version.
comment :: MUR = "Multi-scale Ultra-high Reolution"
license :: These data are available free of charge under data policy of JPL PO.DAAC.
id :: MUR-JPL-L4-GLOB-v04.1
naming_authority :: org.ghrsst
product_version :: 04.1
uuid :: 27665bc0-d5fc-11e1-9b23-0800200c9a66
gds_version_id :: 2.0
netcdf_version_id :: 4.1
date_created :: 20150818T185930Z
start_time :: 20030101T090000Z
stop_time :: 20030101T090000Z
time_coverage_start :: 20021231T210000Z
time_coverage_end :: 20030101T210000Z
file_quality_level :: 1
source :: AMSRE-REMSS, AVHRR_Pathfinder-PFV5.2-NODC_day, AVHRR_Pathfinder-PFV5.2-NODC_night, MODIS_A-JPL, MODIS_T-JPL, iQUAM-NOAA/NESDIS, Ice_Conc-OSISAF
platform :: Aqua, DMSP, NOAA-POES, Suomi-NPP, Terra
sensor :: AMSR-E, AVHRR, MODIS, SSM/I, VIIRS, in-situ
Metadata_Conventions :: Unidata Observation Dataset v1.0
metadata_link :: http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MUR-JPL-L4-GLOB-v04.1
keywords :: Oceans > Ocean Temperature > Sea Surface Temperature
keywords_vocabulary :: NASA Global Change Master Directory (GCMD) Science Keywords
standard_name_vocabulary :: NetCDF Climate and Forecast (CF) Metadata Convention
southernmost_latitude :: -90.0
northernmost_latitude :: 90.0
westernmost_longitude :: -180.0
easternmost_longitude :: 180.0
spatial_resolution :: 0.01 degrees
geospatial_lat_units :: degrees north
geospatial_lat_resolution :: 0.01 degrees
geospatial_lon_units :: degrees east
geospatial_lon_resolution :: 0.01 degrees
acknowledgment :: Please acknowledge the use of these data with the following statement: These data were provided by JPL under support by NASA MEaSUREs program.
creator_name :: JPL MUR SST project
creator_email :: ghrsst@podaac.jpl.nasa.gov
creator_url :: http://mur.jpl.nasa.gov
project :: NASA Making Earth Science Data Records for Use in Research Environments (MEaSUREs) Program
publisher_name :: GHRSST Project Office
publisher_url :: http://www.ghrsst.org
publisher_email :: ghrsst-po@nceo.ac.uk
processing_level :: L4
cdm_data_type :: grid

Download granules for an extended period of time

Since you cannot slice data spatially, downloading granules is going to take up a lot of disk space if you only need data within a small bounding box. For our task, we wrote a simple function to slice data and wrote it into a new file.

You can consult the earthacess library website or their notebooks for code snippets on how to browse and look up collections. For this notebook, we mainly focus on the downloading aspect. First, we need to get the list of granules to download.

Since earthacess does not support spatial slicing, we developed a method to download, slice, combine, and export data yearly, then finally delete temporary downloaded files to save disk space. Assumed that you already knew the temporal, spatial range of the dataset of your chosen, we first download the data by year into a temporary folder, then slice the data and then export the combined data to another folder.

# Our approach
# Month end not included

def download_and_combine_granules(short_name, month_start, month_end, lat1=5, lat2=25, lon1=60, lon2=80):
    for month in pd.date_range(month_start, month_end, freq='M').strftime('%Y-%m-%d'):      
        print('Collecting granules')
        granules = earthaccess.granule_query().short_name(short_name).temporal(f'{month[:7]}-01',month).get(366)
        
        MAIN_FOLDER = 'demonstrated data/earth_data'
        TEMP_FOLDER = 'temp'
        path_temp_folder = os.path.join(MAIN_FOLDER, TEMP_FOLDER)
        path_processed_folder = os.path.join(MAIN_FOLDER, short_name)
        # create folder to store data
        if not os.path.exists(path_temp_folder):
            os.makedirs(path_temp_folder)
        if not os.path.exists(path_processed_folder):
            os.makedirs(path_processed_folder)
        files = earthaccess.download(granules, path_temp_folder)
       
        
        ## if dataset coordinates are slice-able, use:
        print('Slicing...')
        data = xr.open_mfdataset(f'{path_temp_folder}/*.nc').sel(lat=slice(lat1, lat2), lon=slice(lon1, lon2))
    
        
        # combine files together 
        ## for some collections, coordinate names are 'lat' and 'lon' while their underlying indices are 'latitude' and 'longitude', respectively
        ## may or may not be applicable for other datasets on the site.
        ## get bounding box if not sliceable
        ### lat1_idx, lat2_idx, lon1_idx, lon2_idx = get_bounding_box(os.path.join(path_temp_folder, first_file), lat1, lat2, lon1, lon2)
        ### data = xr.open_mfdataset(f'{path_temp_folder}/*.nc').isel(latitude=slice(lat1_idx, lat2_idx+1), longitude=slice(lon1_idx, lon2_idx+1))
        
        data.to_netcdf(f'{path_processed_folder}/{month}.nc')
        
        # delete files in the temporary folder
        print('Deleting temporary files...')
        files = glob.glob(f'{path_temp_folder}/*.*')
        for f in files:
            os.remove(f)

def get_bounding_box(file_path, lat1=0, lat2=30, lon1=60, lon2=80):
    """
    The dataset we experimented did not have indexed coordinates, 
    so we resorted to slicing using index positions
    """
    ds = xr.open_dataset(file_path)
    
    lat_vals = ds.lat.values
    lon_vals = ds.lon.values
    
    lat1_idx = np.abs(lat_vals-lat1).argmin()
    lat2_idx = np.abs(lat_vals-lat2).argmin()
    lon1_idx = np.abs(lon_vals-lon1).argmin()
    lon2_idx = np.abs(lon_vals-lon1).argmin()
    
    return lat1_idx, lat2_idx, lon1_idx, lon2_idx

# download and combine every month's worth of data
download_and_combine_granules(short_name='MUR25-JPL-L4-GLOB-v04.2',
                          month_start='2003-01', month_end='2003-03', 
                          lat1=5, lat2=25, lon1=60, lon2=80)

Collecting granules
 Getting 31 granules, approx download size: 0.0 GB
Slicing...
Deleting temporary files...
Collecting granules
 Getting 28 granules, approx download size: 0.0 GB
Slicing...
Deleting temporary files...

list_of_missing_dates = ["2003-01-04", "2003-01-18", "2003-02-18"]

for date in list_of_missing_dates:
    result = earthaccess.search_data(
        short_name='MUR-JPL-L4-GLOB-v4.1',
        cloud_hosted=True,
        bounding_box = (60, 5, 80, 25),
        temporal=(date, date)
    )
    
    MAIN_FOLDER = 'demonstrated data/earth_data'
    TEMP_FOLDER = 'temp'
    path_temp_folder = os.path.join(MAIN_FOLDER, TEMP_FOLDER)
    file = earthaccess.download(result, path_temp_folder)

Granules found: 1
 Getting 1 granules, approx download size: 0.0 GB
Error while downloading the file 20030218090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/earthaccess/store.py", line 488, in _download_file
    r.raise_for_status()
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20030218090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc

Written by Minh Phan

Authorize credentials

Import necessary libraries

Download data to your local machine using earthaccess library

Streaming granules

Download granules for an extended period of time

Combine files together