Environmental Data Analysis with MATLAB or Python 3rd Edition Lecture 2

Slide Note

Learn about data analysis using MATLAB or Python, covering topics such as probability, linear models, Fourier series, filter theory, factor analysis, hypothesis testing, and more. Explore tools for analyzing data and work with temperature data from the Black Rock Forest. Get hands-on experience in understanding, visualizing, and interpreting environmental data for research and analysis purposes.

vickroy_m Follow

Uploaded on Sep 27, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

Environmental Data Analysis with MATLAB or Python 3rdEdition Lecture 2

SYLLABUS Lecture 01 Lecture 02 Lecture 03 Lecture 04 Lecture 05 Lecture 06 Lecture 07 Lecture 08 Lecture 09 Lecture 10 Lecture 11 Lecture 12 Lecture 13 Lecture 14 Lecture 15 Lecture 16 Lecture 17 Lecture 18 Lecture 19 Lecture 20 Lecture 21 Lecture 22 Lecture 23 Lecture 24 Lecture 25 Lecture 26 Intro; Using MTLAB or Python Looking At Data Probability and Measurement Error Multivariate Distributions Linear Models The Principle of Least Squares Prior Information Solving Generalized Least Squares Problems Fourier Series Complex Fourier Series Lessons Learned from the Fourier Transform Power Spectra Filter Theory Applications of Filters Factor Analysis and Cluster Analysis Empirical Orthogonal functions and Clusters Covariance and Autocorrelation Cross-correlation Smoothing, Correlation and Spectra Coherence; Tapering and Spectral Analysis Interpolation and Gaussian Process Regression Linear Approximations and Non Linear Least Squares Adaptable Approximations with Neural Networks Hypothesis testing Hypothesis Testing continued; F-Tests Confidence Limits of Spectra, Bootstraps

goals of the lecture get you started looking critically at data

Objectives when taking a first look at data Understand the general character of the dataset. Understand the general behavior of individual parameters. Detect obvious problems with the data.

Tools for Looking at Data covered in this lecture reality checks time plots histograms rate information scatter plots

Black Rock Forest Temperature I downloaded the weather station data from the International Research Institute (IRI) for Climate and Society at Lamont-Doherty Earth Observatory, which is the data center used by the Black Rock Forest Consortium for its environmental data. About 20 parameters were available, but I downloaded only hourly averages of temperature. My original file, brf_raw.txt has time in a format that I thought would be hard to work with, so I wrote a MatLab script, brf_convert.m, that converted it into time in days, and wrote the results into the file that I gave you.

format conversion days from start of first year of data calendar date/time 0100-0159 2 Jan 1997 1.042 sequential time variable need for data analysis but format conversions provide opportunity for error to creep into dataset

Reality Checks properties that your experience tells you that the data must have check you expectations against the data

Reality Checks What do you expect the data to look like? hourly measurements thirteen years of data location in New York (moderate climate)

take a moment ... to sketch a plot of what you expect the data to look like

Reality Checks What do you expect the data to look like? hourly measurements time increments by 1/24 day per sample about 24*365*13 = 113880 lines of data temperatures in the -20 to +35 deg C range diurnal and seasonal cycles thirteen years of data location in New York (moderate climate)

Does time increment by 1/24 days per sample? 1/24 = 0.0417 0.00000 0.04167 0.08333 0.12500 0.16667 -17.27000 -17.85000 -18.42000 -18.94000 -19.29000 Yes

Are there about 24*365*20 = 113880 lines of data ? length(D) 110430 Yes, but maybe some data is missing

temperatures in the -20 to +35 deg C range? diurnal and seasonal cycles?

hot spike -20 to +35 range data drop-outs annual cycle cold spikes Temperatures in the -20 to +35 deg C range? Mostly Diurnal and seasonal cycles? Certainly seasonal.

Data Drop-outs common in datasets the instrument wasn t working for a while take two forms: missing rows of table data set to some default value 0 n/a -999 all common

50 days of data from winter 50 days of data from summer diurnal cycle data drop-out cold spike

Histograms determine range of the majority of data values quantifies the frequency of occurrence of data at different data values easy to spot over-represented and under- represented values

MATLAB code for Histogram Lh = 100; dmin = min(d); dmax = max(d); bins = dmin+(dmax-dmin)*[0:Lh-1] /(Lh-1); dhist = hist(d, bins) ;

Python code for Histogram # histogram Lh = 100; # number of bins in histogram dmin = np.min(d); # minimum bin dmax = np.max(d); # maximum bin c, e = np.histogram(d,Lh,(dmin,dmax)); # create histogram Nc = len(c); # lengths of counts Ne = len(e); # length of edges counts = eda_cvec(c); # vector of counts edges = eda_cvec( e[0:Ne-1] ); # vector of edges centers = edges + 0.5*(edges[1,0]-edges[0,0]); # centers

Histogram of Black Rock Forest temperatures counts temperature, C

Alternate ways of displaying a histogram A) B) counts temperature, C

Moving-Window Histograms Series of histograms, each on a relatively short time interval of data Advantage: Shows the way that the frequency of occurrence of data varies with time Disadvantage: Each histogram is computed using less data, and so is less accurate

Moving-Window Histogram of Black Rock Forest temperatures time, days 0 5000 60 temperature, C 0 40

good use of FOR loop MATLAB Version offset=1000; Lw=floor(N/offset)-1; Dhist = zeros(Lh, Lw); for i = [1:Lw]; j=1+(i-1)*offset; k=j+offset-1; Dhist(:,i) = hist(d(j:k), bins)'; end

good use of FOR loop Python Version toffset = 30.0; # offset of 30 days offset = floor(toffset/Dt); # offset in samples Lw = floor( N/offset ); # number of windows Lh = 100; # number of bins in each histogram tend = toffset*Lw; # last time Dw = np.zeros((Lh,Lw)); dmin = -60; dmax = 40; for i in range(Lw): # loop over time windows j=(i-1)*offset; # index of first data in window k=j+offset; # index of last data in window c, e = np.histogram(d[j:k,0],Lh,(dmin,dmax)); Nc = len(c); counts = eda_cvec(c); Dw[0:Lh,i:i+1]=counts;

Rate Information how fast a parameter is changing with time or with distance

finite-difference approximation to derivative

MATLAB code for derivative N=length(d); dddt=(d(2:N)-d(1:N-1))./(t(2:N)-t(1:N-1)); [N, M]=D.shape; dddt = np.divide( d[1:N,0:1]-d[0:N-1,0:1], t[1:N,0:1]-t[0:N-1,0:1] );

hypothetical storm event note that more time has negative dd/dt discharge, cfs d/dt discharge, cfs / day 0 500 1000 -500 0 500 0 0 rain 1 1 2 2 3 3 4 4 draining of land time, days time, days 5 5 6 6 7 7 8 8 9 9 10 10

Hypothesis rate of change in discharge correlates with amount of discharge logic a river is bigger when it has high discharge a big river flows faster than a small river a river that flows faster drains away water faster (might only be true after the rain has stopped)

MATLAB Script purpose: make two separate plots, one for times of increasing discharge, one for times of decreasing discharge pos = find(dddt>0); neg = find(dddt<0); - - - plot(d(pos),dddt(pos),'k.'); - - - plot(d(neg),dddt(neg),'k.');

Python version r = np.where( dddt>=0 ); pos = r[0]; r = np.where( dddt<0 ); neg = r[0]; dpos=d[pos,0:1]; dddtpos=dddt[pos,0:1]; dneg=d[neg,0:1]; dddtneg=dddt[neg,0:1]; . . . plt.plot(dpos,dddtpos,'k. ); . . . plt.plot(dneg,dddtneg,'k.');

Atlantic Rock Dataset I downloaded rock chemistry data from PetDB s website at www.petdb.org. Their database contains chemical information about ocean floor igneous and metamorphic rocks. I extracted all samples from the Atlantic Ocean that had the following chemical species: SiO2, TiO2, Al2O3, FeOtotal, MgO, CaO, Na2O and K2O My original file, rocks_raw.txt included a description of the rock samples, their geographic location and other textual information. However, I deleted everything except the chemical data from the file, rocks.txt, so it would be easy to read into MatLab. The order of the columns is as is given above and the units are weight percent.