Problem of creating forcing file with Pyroms(webob.exc.HTTPError: 302 Found)

General scientific issues regarding ROMS

Moderators: arango, robertson

Post Reply
Message
Author
Zeng
Posts: 24
Joined: Mon Sep 19, 2022 1:06 pm
Location: Sun Yat-Sen University

Problem of creating forcing file with Pyroms(webob.exc.HTTPError: 302 Found)

#1 Unread post by Zeng »

Hello,everyone! I am a beginer of ROMS, and when I followed the videos to learn how to creat forcing files with Pyroms, I encountered one problem.
(base) zeng@zeng-virtual-machine:~/soft/pyroms_soft/pyroms/MERRA-2$ python3 get_MERRA_albedo_from_nasa_opendap_daily.py 2022
url https://goldsmr4.gesdisc.eosdis.nasa.go ... 220101.nc4
Traceback (most recent call last):
File "/home/zeng/soft/pyroms_soft/pyroms/MERRA-2/get_MERRA_albedo_from_nasa_opendap_daily.py", line 57, in <module>
lon = dataset['lon'][:]
File "/home/zeng/anaconda3/lib/python3.9/site-packages/pydap/model.py", line 320, in __getitem__
out.data = self._get_data_index(index)
File "/home/zeng/anaconda3/lib/python3.9/site-packages/pydap/model.py", line 349, in _get_data_index
return self._data[index]
File "/home/zeng/anaconda3/lib/python3.9/site-packages/pydap/handlers/dap.py", line 142, in __getitem__
raise_for_status(r)
File "/home/zeng/anaconda3/lib/python3.9/site-packages/pydap/net.py", line 33, in raise_for_status
raise HTTPError(
webob.exc.HTTPError: 302 Found
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://urs.earthdata.nasa.gov/oauth/au ... re</a>.</p>
</body></html>


I guess there is something wrong with URL?? After I logged into the website, I found that the DATA URL (https://goldsmr4.gesdisc.eosdis.nasa.go ... 220101.nc4) is consistent with the URL created by the code.. So I cannot find the problems, is there anyone willing to help me? Thank you sooooo much! :D

The code is shown below:

Code: Select all

import matplotlib
matplotlib.use('Agg')
import numpy as np
import netCDF4
from datetime import datetime
import pyroms
import pyroms_toolbox

#import http.cookiejar
#import netrc
#import urllib.request, urllib.error, urllib.parse
#import re
#import pydap.lib
#from pydap.exceptions import ClientError
#import logging
import sys
from pydap.client import open_url
from pydap.cas.urs import setup_session

year = int(sys.argv[1])

invarname = 'ALBEDO'

outvarname = 'albedo'
outtimename = 'albedo_time'

server = 'https://goldsmr4.gesdisc.eosdis.nasa.gov'

leap = year%4
if leap == 0:
    daysinmonth = ([31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31])
    daysinyear = 366
else:
    daysinmonth = ([31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31])
    daysinyear = 365

#if (year <= 1992):
#    file_tag = 'MERRA101'
#elif ((year >= 1993) & (year <= 2000)):
#    file_tag = 'MERRA201'
#elif ((year >= 2001) & (year <= 2009)):
#    file_tag = 'MERRA301'
#elif (year >= 2010):
#    file_tag = 'MERRA300'
file_tag = 'MERRA2_400'

#read grid and variable attributes from the first file
year_tag = '%04d' %year
month_tag = '01'
day_tag = '01'
date_tag = year_tag + month_tag + day_tag
url = server + '/opendap/MERRA2/M2T1NXRAD.5.12.4/' + year_tag + '/' + month_tag + '/' + \
      file_tag + '.tavg1_2d_rad_Nx.' + date_tag + '.nc4'
print('url',url)  
session = setup_session("username","password",url)
dataset = open_url(url, session =session)
lon = dataset['lon'][:]

#shift data between 0 and 360 deg.
gidx = np.where(np.abs(lon) < 1.0e-10)[0][0]
lon = np.asarray(lon)
lon = lon + 180
lat = dataset['lat'][:]
lat = np.asarray(lat)
spval = dataset[invarname].missing_value
units = dataset[invarname].units
long_name = dataset[invarname].long_name

#get data from NASA opendap
for month in range(1-1,1):
    nday = 0

    #create ROMS forcing file
    month_tag = '%02d' %(month+1)
    outfile = 'Forcings/MERRA_' + outvarname + '_3hours_' + year_tag + '_' + month_tag + '.nc'
    nc = netCDF4.Dataset(outfile, 'w', format='NETCDF3_64BIT')
    nc.Author = sys._getframe().f_code.co_name
    nc.Created = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    nc.title = 'MERRA-2 dataset. Modern Era Retrospective-analysis'

    nc.createDimension('lon', np.size(lon))
    nc.createDimension('lat', np.size(lat))
    nc.createDimension(outtimename, None)

    nc.createVariable('lon', 'f8', ('lon'))
    nc.variables['lon'].long_name = 'longitude'
    nc.variables['lon'].units = 'degrees_east'
    nc.variables['lon'][:] = lon

    nc.createVariable('lat', 'f8', ('lat'))
    nc.variables['lat'].long_name = 'latitude'
    nc.variables['lat'].units = 'degrees_north'
    nc.variables['lat'][:] = lat

    nc.createVariable(outtimename, 'f8', (outtimename))
    nc.variables[outtimename].units = 'days since 1900-01-01 00:00:00'
    nc.variables[outtimename].calendar = 'LEAP'
    dstart = pyroms_toolbox.date2jday(datetime(year, month+1, 1, 0, 0))
    roms_time = np.arange(dstart, dstart+daysinmonth[month], 3./24)
    nc.variables[outtimename][:] = roms_time

    nc.createVariable(outvarname, 'f', (outtimename, 'lat', 'lon'), fill_value=spval)
    nc.variables[outvarname].long_name = long_name
    nc.variables[outvarname].units = units
    nc.variables[outvarname].coordinates = 'lon lat'

    for day in range(1):
#    if year == 2010:
#      if ((month+1 >= 6) & (month+1 <= 8)):
#        file_tag = 'MERRA301'
#      else:
#        file_tag = 'MERRA300'
        day_tag = '%02d' %(day+1)
        date_tag = year_tag + month_tag + day_tag
        url = server + '/opendap/MERRA2/M2T1NXRAD.5.12.4/' + year_tag + '/' + month_tag + '/' + \
              file_tag + '.tavg1_2d_rad_Nx.' + date_tag + '.nc4'
        session = setup_session("username","password",url)
        dataset = open_url(url, session =session)
        var = dataset[invarname][:]
        var = np.asarray(var)
        #shift data between 0 and 360 deg.
        svar = np.zeros(var.shape)
        svar[:,:,:len(lon)-gidx] = var[:,:,gidx:]
        svar[:,:,len(lon)-gidx:] = var[:,:,:gidx]
        mask = np.invert(np.ma.masked_values(svar, spval).mask).astype('float') #mask
        np.seterr(divide='ignore', invalid='ignore') #to ignorewarning using missing value for calculation e.g (1.0 or 1/NA)
        var_daily = (svar * mask).sum(axis=0) /  mask.sum(axis=0)
        np.seterr(divide='warn', invalid='warn') #to reactivate warning using missing value for calculation
        idx = np.where(np.isnan(var_daily) == True)
        var_daily[idx] = spval
        var_daily = np.ma.masked_values(var_daily, spval)
        nc.variables[outvarname][nday,:,:] = var_daily
        nday = nday + 1
#       dataset.close()

    nc.close()
Last edited by Zeng on Sun Feb 12, 2023 2:54 am, edited 1 time in total.

Zeng
Posts: 24
Joined: Mon Sep 19, 2022 1:06 pm
Location: Sun Yat-Sen University

Re: Problem of creating forcing file with Pyroms(webob.exc.HTTPError: 302 Found)

#2 Unread post by Zeng »

Sorry, it seems like that the URL cannot be displayed completely.. In fact, the URL created by the code is exactly the same as the URL displayed on the data website (url https:// goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2/M2T1NXRAD.5.12.4/2022/01/MERRA2_400.tavg1_2d_rad_Nx.20220101.nc4). :cry: :cry:

abiola
Posts: 6
Joined: Sat Dec 03, 2022 5:05 pm
Location: Florida Atlantic University

Re: Problem of creating forcing file with Pyroms(webob.exc.HTTPError: 302 Found)

#3 Unread post by abiola »

Hi Zeng,

You first need to register and activate your account on the data site. Go to https://urs.earthdata.nasa.gov/profile, click on applications, and check your authorized apps. You need to activate or authorize if you are yet to be authorized to use GESDISC and GESDISC Test Data Archive.

I hope this works for you.

Zeng
Posts: 24
Joined: Mon Sep 19, 2022 1:06 pm
Location: Sun Yat-Sen University

Re: Problem of creating forcing file with Pyroms(webob.exc.HTTPError: 302 Found)

#4 Unread post by Zeng »

abiola wrote: Wed Feb 08, 2023 8:28 pm Hi Zeng,

You first need to register and activate your account on the data site. Go to https://urs.earthdata.nasa.gov/profile, click on applications, and check your authorized apps. You need to activate or authorize if you are yet to be authorized to use GESDISC and GESDISC Test Data Archive.

I hope this works for you.
Hello! abiola! Thanks for your reply, when I complete the authorization of NASA GESDISC DATA ARCHIVE, it works! Thanks again for your attention and kindly advice! You are absolutely the angel of my research career!! :lol:

when I re-run the program, one line of screen feedback is: socket.timeout: The read operation timed out , so I add the timeout to the code of "dataset = open_url(url, session =session)", and the modified code is "dataset = open_url(url, session =session, timeout=10)". I want to check whether it works, but when I re-run it, there is another error..

Code: Select all

(base) zeng@zeng-virtual-machine:~/soft/pyroms_soft/pyroms/MERRA-2$ Traceback (most recent call last):
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/zeng/anaconda3/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/zeng/anaconda3/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/zeng/anaconda3/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/zeng/anaconda3/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/home/zeng/anaconda3/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7faee1a2b9a0>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='goldsmr4.gesdisc.eosdis.nasa.gov', port=80): Max retries exceeded with url: /data-redirect?code=f76711b8b62f137bf140cc907809c1f297c925fbbe3e6a0a3e4cc2d651b0e0b5&state=aHR0cHM6Ly9nb2xkc21yNC5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9vcGVuZGFwL01FUlJBMi9NMlQxTlhSQUQuNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC50YXZnMV8yZF9yYWRfTnguMjAyMjAxMDEubmM0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7faee1a2b9a0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zeng/soft/pyroms_soft/pyroms/MERRA-2/get_MERRA_albedo_from_nasa_opendap_daily.py", line 117, in <module>
    session = setup_session("enzo999","Zjy950324",url)
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/pydap/cas/urs.py", line 14, in setup_session
    session = get_cookies.setup_session('https://urs.earthdata.nasa.gov',
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/pydap/cas/get_cookies.py", line 91, in setup_session
    res = session.get(check_url, auth=(username, password))
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 723, in send
    history = [resp for resp in gen]
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 723, in <listcomp>
    history = [resp for resp in gen]
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 266, in resolve_redirects
    resp = self.send(
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/zeng/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='goldsmr4.gesdisc.eosdis.nasa.gov', port=80): Max retries exceeded with url: /data-redirect?code=f76711b8b62f137bf140cc907809c1f297c925fbbe3e6a0a3e4cc2d651b0e0b5&state=aHR0cHM6Ly9nb2xkc21yNC5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9vcGVuZGFwL01FUlJBMi9NMlQxTlhSQUQuNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC50YXZnMV8yZF9yYWRfTnguMjAyMjAxMDEubmM0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7faee1a2b9a0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
^C
[1]+  退出 1                python3 get_MERRA_albedo_from_nasa_opendap_daily.py 2022
I guess the server shut down the data transmition channel? I am currently trying to solve this problem. And I would realy appreciate it if you knew how to solve it! :mrgreen:

abiola
Posts: 6
Joined: Sat Dec 03, 2022 5:05 pm
Location: Florida Atlantic University

Re: Problem of creating forcing file with Pyroms(webob.exc.HTTPError: 302 Found)

#5 Unread post by abiola »

Hi Zeng,

I am not sure if the timeout you added is necessary, though, or probably it is necessary for python3, but I don't think so. You might remove the timeout and try again. Sometimes, it shows some error, especially if you go in bulk. You can try each file once. If it doesn't work still, you might find python2 worthy to help you through as it poses fewer complications...

Zeng
Posts: 24
Joined: Mon Sep 19, 2022 1:06 pm
Location: Sun Yat-Sen University

Re: Problem of creating forcing file with Pyroms(webob.exc.HTTPError: 302 Found)

#6 Unread post by Zeng »

abiola wrote: Fri Feb 10, 2023 4:25 pm Hi Zeng,

I am not sure if the timeout you added is necessary, though, or probably it is necessary for python3, but I don't think so. You might remove the timeout and try again. Sometimes, it shows some error, especially if you go in bulk. You can try each file once. If it doesn't work still, you might find python2 worthy to help you through as it poses fewer complications...
Dear abiola, thank you for your kind reply. I will try it.. :wink: Nice to meet you, let's keep in touch! :mrgreen:

Post Reply