Is there a better way to calculate trend lines on periodic functions?

  • 6 Replies
  • 4491 Views

0 Members and 1 Guest are viewing this topic.

*

Offline CliffordK

  • Neilep Level Member
  • ******
  • 6321
  • Site Moderator
    • View Profile
Actually looking at some of the environmental information.
It seems as if there is a problem with poor fit of trend lines on periodic, or partially periodic functions.

I think some kind of a least-squares trend line is typically used.  These calculations are based on the Slope() and Intercept() functions in OpenOffice.

To illustrate some of the problems, here are a couple of basic functions.

Plotting a trend line to a sine wave.
[attachment=14107]

If I plotted for 0 to 1800 degrees (5 complete cycles), starting on a peak and ending on a valley, I got a really bad match (orange line)
I got a reasonably good match for the slope, but was off on the average/intercept if I plotted from 0 to 1620 (5 peaks, 4 valleys), or 180 to 1800 (4 peaks, 5 valleys).
The best match was plotting from peak to peak or valley to valley.  (90 to 1530, or 270 to 1710).

Anyway, so choosing the end points is very important.

Then, I decided to try a sawtooth function (up slope, 10/8, down slope -10/2)
[attachment=14105]

It turns out that it is very hard to get a good match for the trend line (which should have 0 slope, average of 5).

With the exception of starting on a peak and ending on a valley, the majority of the trend lines have a positive slope.

I suppose my problem is that if can't match a trend line to a simple geometric function...  how can I trust matching it to natural systems.

Suggestions?

*

Offline Bored chemist

  • Neilep Level Member
  • ******
  • 8736
    • View Profile
Do you know about Fourier analysis?
http://en.wikipedia.org/wiki/Fourier_transform
Please disregard all previous signatures.

*

Offline CliffordK

  • Neilep Level Member
  • ******
  • 6321
  • Site Moderator
    • View Profile
Do you know about Fourier analysis?
http://en.wikipedia.org/wiki/Fourier_transform
Thanks,

Yes, I know about the FFT...  at least in theory.
Perhaps I should look at it some more.

What I'm actually looking at is the Temperature Trends.  In particular, 1997/1998 to present.

[attachment=14120]

As I noted from the sine curve above, one can obviously diddle with the end-points to prove anything that one wants.

Visually looking at the graph, it appears to have no significant trend. 

But, when I plot the trend line, it invariably gives me a slight positive slope [???]

What I noticed was that the temperatures from 1999 to 2002 and 2008 to 2010 seem to increase in a sawtooth pattern, with a sharp downswing, and a gradual upswing, thus weighting more negative towards the early years, and more positive towards the later years.

And, what I was able to demonstrate with the stylized sawtooth pattern is that it will cause an upward trend in the trend  line in most cases except when one begins with a peak and ends with a valley.

If I was looking at something with a slope of 1:2 or so...  I really wouldn't care much about a fractional difference.  But, with a nearly flat slope, it makes a huge difference.  [xx(]

Anyway, it doesn't seem like a FFT type of function, but perhaps it will give me additional analysis tools and it certainly is something that I need to understand better.

*

Offline Geezer

  • Neilep Level Member
  • ******
  • 8328
  • "Vive la résistance!"
    • View Profile
I think BC is right. Try breaking it down into frequencies and see what the relative amplitudes look like. I have a suspicion that we humans can "eyeball" just about anything that takes our fancy from complex waveforms. (I've done a lot of that.) 
There ain'ta no sanity clause, and there ain'ta no centrifugal force ćther.

*

Offline Bored chemist

  • Neilep Level Member
  • ******
  • 8736
    • View Profile
If it was a sawtooth then an FFT (or any other FT) would show it very clearly.
I don't think you can say it's real on that data.
Please disregard all previous signatures.

*

Offline imatfaal

  • Neilep Level Member
  • ******
  • 2787
  • rouge moderator
    • View Profile
Cliff - BC or one of the other practising scientists can guide you on this but IIRC a regression coefficient R^2 of 0.004 is far too low to allow the correlation to be claimed.  For a simple linear regression R^2 ranges from 1 to 0; with 1 being a complete correlation (ie y=ax+b) and 0 being no correlation. 

This is quite apart from the fact that linear regression is the wrong tool - but is does explain why you were mystified to see the slope, it's only just evident by the slightest of margins! 
There’s no sense in being precise when you don’t even know what you’re talking about.  John Von Neumann

At the surface, we may appear as intellects, helpful people, friendly staff or protectors of the interwebs. Deep down inside, we're all trolls. CaptainPanic @ sf.n

*

Offline Bored chemist

  • Neilep Level Member
  • ******
  • 8736
    • View Profile
A full statistical analysis s a lot of work, but there's one simple thing you can do.
The value of R^2 gives you the fraction of the variation that is accounted for by the "model". So, in this case, you are assuming a model that says the data fits to y=mx+c and has noise that explains why it doesn't all lie exactly on the line.
In this case, the model explains 0.0049 of the variation in temperature and the other 99.5% of the variability is caused by something else.
 
As Imatfaal says, that's pretty hopeless.
Please disregard all previous signatures.