The Naked Scientists

The Naked Scientists Forum

Author Topic: Is there a better way to calculate trend lines on periodic functions?  (Read 4446 times)

Offline CliffordK

  • Neilep Level Member
  • ******
  • Posts: 6321
  • Thanked: 3 times
  • Site Moderator
    • View Profile
Actually looking at some of the environmental information.
It seems as if there is a problem with poor fit of trend lines on periodic, or partially periodic functions.

I think some kind of a least-squares trend line is typically used.  These calculations are based on the Slope() and Intercept() functions in OpenOffice.

To illustrate some of the problems, here are a couple of basic functions.

Plotting a trend line to a sine wave.


If I plotted for 0 to 1800 degrees (5 complete cycles), starting on a peak and ending on a valley, I got a really bad match (orange line)
I got a reasonably good match for the slope, but was off on the average/intercept if I plotted from 0 to 1620 (5 peaks, 4 valleys), or 180 to 1800 (4 peaks, 5 valleys).
The best match was plotting from peak to peak or valley to valley.  (90 to 1530, or 270 to 1710).

Anyway, so choosing the end points is very important.

Then, I decided to try a sawtooth function (up slope, 10/8, down slope -10/2)


It turns out that it is very hard to get a good match for the trend line (which should have 0 slope, average of 5).

With the exception of starting on a peak and ending on a valley, the majority of the trend lines have a positive slope.

I suppose my problem is that if can't match a trend line to a simple geometric function...  how can I trust matching it to natural systems.

Suggestions?


 

Offline Bored chemist

  • Neilep Level Member
  • ******
  • Posts: 8669
  • Thanked: 42 times
    • View Profile
Do you know about Fourier analysis?
http://en.wikipedia.org/wiki/Fourier_transform
 

Offline CliffordK

  • Neilep Level Member
  • ******
  • Posts: 6321
  • Thanked: 3 times
  • Site Moderator
    • View Profile
Do you know about Fourier analysis?
http://en.wikipedia.org/wiki/Fourier_transform
Thanks,

Yes, I know about the FFT...  at least in theory.
Perhaps I should look at it some more.

What I'm actually looking at is the Temperature Trends.  In particular, 1997/1998 to present.



As I noted from the sine curve above, one can obviously diddle with the end-points to prove anything that one wants.

Visually looking at the graph, it appears to have no significant trend. 

But, when I plot the trend line, it invariably gives me a slight positive slope ???

What I noticed was that the temperatures from 1999 to 2002 and 2008 to 2010 seem to increase in a sawtooth pattern, with a sharp downswing, and a gradual upswing, thus weighting more negative towards the early years, and more positive towards the later years.

And, what I was able to demonstrate with the stylized sawtooth pattern is that it will cause an upward trend in the trend  line in most cases except when one begins with a peak and ends with a valley.

If I was looking at something with a slope of 1:2 or so...  I really wouldn't care much about a fractional difference.  But, with a nearly flat slope, it makes a huge difference.  [xx(]

Anyway, it doesn't seem like a FFT type of function, but perhaps it will give me additional analysis tools and it certainly is something that I need to understand better.
 

Offline Geezer

  • Neilep Level Member
  • ******
  • Posts: 8328
  • "Vive la résistance!"
    • View Profile
I think BC is right. Try breaking it down into frequencies and see what the relative amplitudes look like. I have a suspicion that we humans can "eyeball" just about anything that takes our fancy from complex waveforms. (I've done a lot of that.) 
 

Offline Bored chemist

  • Neilep Level Member
  • ******
  • Posts: 8669
  • Thanked: 42 times
    • View Profile
If it was a sawtooth then an FFT (or any other FT) would show it very clearly.
I don't think you can say it's real on that data.
 

Offline imatfaal

  • Neilep Level Member
  • ******
  • Posts: 2787
  • rouge moderator
    • View Profile
Cliff - BC or one of the other practising scientists can guide you on this but IIRC a regression coefficient R^2 of 0.004 is far too low to allow the correlation to be claimed.  For a simple linear regression R^2 ranges from 1 to 0; with 1 being a complete correlation (ie y=ax+b) and 0 being no correlation. 

This is quite apart from the fact that linear regression is the wrong tool - but is does explain why you were mystified to see the slope, it's only just evident by the slightest of margins! 
 

Offline Bored chemist

  • Neilep Level Member
  • ******
  • Posts: 8669
  • Thanked: 42 times
    • View Profile
A full statistical analysis s a lot of work, but there's one simple thing you can do.
The value of R^2 gives you the fraction of the variation that is accounted for by the "model". So, in this case, you are assuming a model that says the data fits to y=mx+c and has noise that explains why it doesn't all lie exactly on the line.
In this case, the model explains 0.0049 of the variation in temperature and the other 99.5% of the variability is caused by something else.
 
As Imatfaal says, that's pretty hopeless.
 

The Naked Scientists Forum


 

SMF 2.0.10 | SMF © 2015, Simple Machines
SMFAds for Free Forums