Where Do I Get Forex Historical Data?

I've had a bunch of inquiries on where I obtain my forex historical data so I figured it's best to address it right here.  

In my 2+ years of backtesting experience, I've found a variety of forex historical data providers.  Some have been good, some bad.  I have only paid for historical data once and it was cheap, $134.99 cheap.  The major issue with this data is that it's indicative and has holes.  For those of you that don't know what indicative data is, it's data compiled from multiple providers so the prices could reflect either the median or average price.  This is not a true reflection of what's going on when you're trading.  Indicative data in my opinion is useless.  Unfortunately, you'll find this mostly in Foreign Exchange since their is no centralized exchange.  If you're really interested in this data, you can find it at http://disktrading.is99.com.

The ultimate goal is to find forex data that is not indicative.  A decent source is at Gain Capital where you can download historical tic data from 2000-present.  It's a whole lot of data since it's tic but it's not indicative so it's useful.  The major issue is that it's going to take you a long time to download (Gain throttles bandwidth I believe) and clean it up.  This is from someone I received an email from today but his issues were the following:

1. Some of the files were stored in 16bit unicode and not UTF8
2. Spans of data overlaps
3. 2005 is possibly missing chunks
4. Somewhere in 2006, the CSV started using quotes at random which broke his parser

Like I said, it's going to take you a lot of time to get this data in a usable format but it's not bad.  You can find it at ratedata.gaincapital.com (Gain capital is forex.com)

Another decent provider of free forex historical data is at Forexite.  This data is also not indicative and spans back to 2001.  It's 1-minute OHLC data (Open, High, Low, Close.)  The only minor issue with this data is that it's in GMT +1.  Your backtesting has to take this into consideration.  All I did was write a script to convert the time to EST once it was imported into the database.  The time zone differences can get confusing sometimes and I didn't want this to get in the way of my actual backtesting logic.  Interested?  You can find this data at http://www.forexite.com/free_forex_quotes/forex_history_arhiv.html.

Another option is to export the data right out of Metatrader from whatever broker you connect to.  The only issue with this is that the intraday data doesn't go back more than 2 years at the most. 

Once I have all the forexite data imported into a database for the 17 different currency pairs, I'd think about offering it out to the public for download but this wouldn't be until after it's been checked for holes.  

Popularity: 5%

New Forex Trading System

I was busy this weekend doing backtesting.  I focused primarily on backtesting the H-system to improve on its performance.  I was able to obtain much better results by letting the third lot position run which is pretty much what I thought even before backtesting.  I was also able to obtain better results by using a breakeven stop.  My initial stop-loss is always set to 30 pips and I found that moving the stop-loss to breakeven when the position was up 25 pips was optimal.  I know this is only 5 pips from the target but nevertheless, this produced better results than without a breakeven stop or with a breakeven stop lower than 25 pips.  The GBP/USD is a mover and it needs some room to move.  From my experience, it's tough to find profitability with a stop lower than 30 pips.

In the process of coding the backtesting of the H-system, I created a new system that is basically a derivative of the H-system.  This new system isn't difficult to backtest manually so I did it for 2 years but I decided to automate it too just in case I made human errors.  The results from manually and automatically testing were very similar.  This new system is actually more profitable than the H-system on paper.  Like I said over the weekend, I don't want to jump to any final conclusions but by developing a standard backtesting infrastructure, the mistakes that I may have encountered before are lessened.  Therefore, I'm pretty confident with the results.  Now it's time to forward test this new strategy.  My weakness or possible downfall may be my willingness to forward test with real money.  I decided to do this with this new system today, profiting 21 pips on 1 lot.  Here is the equity curve of my newest system from January 2005 to present making the assumption that I'm starting with $10,000.  The label "Profit in USD" in the graph below is wrong.  It should read, "Equity in USD."

 

 

 

 

 

 

 

 

 

 

 

Popularity: 3%

Building Up Historical Forex Data

{mosimage}
In the last couple of weeks, I've been able to build up a pretty extensive collection of Forex historical data.  Included will more than 5 years of historical data for over 15 currency pairs.  The time frames include tick, 1 minute, 5 minute, 15 minute, 30 minute, 60 minute, 180 minute, 240 minute, and daily.  The most intensive and extensive time frame was obviously the tick.  I hope to actually utilize this data in the upcoming days.  Unfortunately all of this is happening on my laptop, where the CPU has been pegged for days turning the laptop into an instant heat source… 

Popularity: 4%

Forex Profit Loss Index

Continuing the spotlight on backtesting, there are 3 different indexes that can be calculated in your backtested results or your live trading results to give an indication of the chance of success.  I mentioned 1 last week, Buy and Hold.  The other 2 are the risk/reward index and the profit/loss index.  

Calculating the profit/loss index gives you a general idea of the success your trading or trading system is having.  A simple example is best suited to illustrate this. 

1. Calculate all of your winning trades in dollar amount.  For this example, I had 5 profitable trades, each $90 for a total of $450.  
2. Calculate all of your losing trading in dollar amount.  For this example, I had 7 unprofitable trades with the following losses: $ -45, -30, -110, -25, -30, -90, and -55.  The total losses amount to $385.
3. Subtract the losing dollar amount from the winning ($450 - $385 = $65)
4. Profit/Loss index =  ((Winning trades + Losing trades)  ÷ Winning trades) * 100

So the profit/loss index for my example would be calculated as such: ($65 ÷ $450) * 100 = 14.4 %

The profit/loss index for this example is 14.4%.  The following key translates these results into something more meaningful:

+100          High Profit/No Losses

+50            Profits > Losses 

0                Profits = Losses

-50             Profits < Losses

-100           No Profits/High Losses

Using this key, the above example would be on the lower end of the 0 - +50 range, not great.  Ideally, you want the index as close to 100% as possible.  If your index is negative, you're losing more than you make.  If your index is 100%, your perfect.

On the surface, this calculation can seem kind of useless but what it does best is give a clear meaning to all of your profits and all of your losses.  How would you otherwise interpret a number like $65?  Yes, it's a profit but in the example case, not a very good profit.

Popularity: 4%

Lots and Lots of Forex Data

A lot of my time in the last week has been dedicated to compiling
backtesting data for multiple currency pairs and multiple time frames. 
I previously had GBP tick data that I was working from but now I'm
collating this data to multiple time frames and increasing my currency
pairs to 35 or so.  This process takes a lot of time and requires some
of the following steps:

1. Creation of databases and tables.  I believe MySQL can handle
   over 240,000,000 rows per database.   When you're dealing with tick
   data, size has to be a consideration.  A couple of years of GBP data
   can easily include over 20,000,000 ticks.
2. Importing the data into respective database tables.
3. Adding indexes for faster querying. Indexes on the datetime are a
   must.  Without indexes, it would be like searching for a term in a book
   without an index.  You would have to go page by page to look for the
   term.  With an index, you look up the term and it points you to the
   page number similar to what a database index does. 
4. Go tick by tick and verify that the data is there.  The key here
   is to verify that there are no gaps in the data.  A gap in the data can
   throw a calculation totally out of whack.

My point of explaining all of this is that there are so many
variables that go into backtesting, the above steps being those that
are abstracted from the user if you are using a given software
package.  With that said, different brokers have different data.  Some
brokers open earlier on certain days and holidays, other brokers close
on certain holidays; the bottom line is that the data is disparate. 
Forex doesn't have a central exchange so the prices you're working from
could be very different from the next person.  In turn, your stochastic
indicator could be showing oversold while another's guys may be at 50. 
For example, right now, my Metatrader charts (Alpari) show the 62 EMA
on the 30-min GBP/USD at 1.9749.  The 62 EMA on the 30-min GBP/USD at
Oanda is 1.9759.  That's a 10 pip difference. 

I've always said
that backtesting is necessary but I do believe that as the time frames
get shorter and shorter, the results have to be taken with more "grains
of salt".  That is why in addition to backtesting your system, it
should always be forward tested.  The object of backtesting should be
to get a general idea of its viability and to assist in determining if
further testing of this trading system is worthwhile.  For example, if
you were to backtest a moving average crossover, it becomes quite
evident that this type of trading system just isn't profitable (most of
the time.)  The negative results associated with this type of system
are just stacked up so high that you can make the assumption that this
system just wouldn't work in real-time.  This of course may not always
be the case.  Interpreting backtesting results requires you to make
assumptions and educated guesses.  Backtesting assists in moving your
system system testing to the next phase whatever this may be.  After
all, we have to make due with what we have and if backtesting can add
the inner confidence to stick to a trading system, then it has helped
in one way or another.  

Popularity: 3%

Developing a systematic approach to trading

After days and days of backtesting, I think I have found a simple system that works for me.  I backtested it over the last 5-6 months on the EUR/USD 60-minute charts.  Of course I understand that there are no guarantees that this system will work well in the future too.  But it does feel really good to have designed a system that I can use going forward.  My goal over the next month or so is to trade religiously using this system and being the gambler I am do so using my live trading account.  It actually feels really great to have a system that is going to generate long, short, cover, and sell signals right on my charts.  Here are my backtested results:

 

 

 

 

 

Popularity: 3%

Week 14 Performance

As I mentioned a couple of days ago, this week was especially hard on me.  I lost $3000 to drop my 14 week earnings to +2500.  There are a couple of major adjustments I’m going to make going into week 15.

1.    Finish a preliminary test plan that at least details 1 trade setup

2.    Stop trading multiple currency pairs.  I’ve been trading and trying to study about 8 different currency pairs.  I feel like this is definately too much for me to handle and most would not recommend a beginner to do this.  I will try to study and trade the EUR/USD ONLY since it is considered to have the most volume.  I will slowly try to incorporate some more of the major pairs into my trading plan as I get more comfortable with the EUR/USD.

3.    Back test, back test, back test.  A couple of months ago I was really interested in back testing trading strategies but lost interest after several attempts proved to be very unprofitable.  I have since started using Metastock with esignal to backtest moving averages.  Metastock has an optimizer function that will go through all combinations of moving averages and return the most profitable combinations.  My goal will be to first to find the most profitable moving averages for the EUR/USD.  I will try to correlate the moving average pairs to produce the most profitable 3 moving averages. 

I’ve already started performing #3 and I currently have Metastock performing backtesting on over a years worth of 1 hour interval data.  It’s very CPU intensive and will take about 4 hours.  It’s currently half way done and so far the best EMA pair is the 7/75.  The 7/75 EMA’s actually have some impressive results over the last year on the 1 hour EUR/USD chart. 

Here is an exact description of the backtest:

Buys/sells when a moving average of 7 periods goes above/below a moving average of  75 periods. 

I also have it factoring in the broker fee and have set the results to buy/sell 1 lot.

Here are the results:

70 profitable trades with average profit of $355.86 per trade.
Highest profit was $1020 and the most consecutive profitable trades was 12 

15 unprofitable trades with average loss of $972.67 per trade.
Highest loss was $2180 and the most consecutive losing trades was only 2 

Over 1 year, if you followed this trading plan by buying or selling 1 lot each trade, you would have profited $10320.00. 

Popularity: 4%

Forex Blog : Stochastic Price Reverse Backtesting

I backtested the Stochastic Price Reverse trading system from 2003-2005 for EUR/USD.  If you want to read more about it, go to http://www.wealth-lab.com.

I have graphed the results and you can view them here.
You can view the actual buy/sell signal results here. 

To quickly summarize, if you were to use this trading system, over the last 3 years, your would have been most successful if you used 6,7, or 8 as %K values. 

 

Popularity: 3%

wealth-lab back testing

I’m sure some of you have already checked out www.wealth-lab.com which is an interactive development laboratory.  I cannot use the developer because it is only available to users outside of the United States and Canada. 

What I have checked out is the ChartScripts that are presented by other users on the site.  You can interactively see how certain equities have performed over the last 5 years using a particular trading system.

One trading system in particular I’ve looked at is the Stochastic Price Reverse - market V4.5.  I’ve looked at the code associated with this to try to emulate this test in my backtesting lab.  These scripts are developed with equities in mind so some of the constants need to be changed.  In my initial runs, I’ve seen a lot of promise with this trading system.  Quite simply, if Stochastic K% is > 97 using 16 period parameter, initiate a sell.  Exit the position after 1 full period.  If stochastic K% is < 5 using 9 period parameter, initiate a buy.  Exit the position after 1 full period.  In addition, this trading system verifies that past close of equity is greater than or less than other past closes.  These constants cannot be used for currencies because of the price change percentages being much lower.  Running this on the 4 major currency pairs, I’ve seen an average 2 year gain of about 1500 pips. 

They use their own proprietary programming language but it is a very readable programming language and layman may be able to understand it.  I’ve posted the code here:

 var  STOCHPANE, BAR: integer;
var  StochPeriod, StochPeriodShort: integer;

{Stoch. period for longs}
StochPeriod:= 9;

{Stoch. period for shorts}
StochPeriodShort:= 16;

for Bar := 20 to BarCount - 1 do
begin
    if not LastPositionActive then
    {Entry rules}
     begin
       if (StochK( Bar, StochPeriod) < 5)  then
         if (0.96*@#AverageC[ Bar-2 ] > @#AverageC[ Bar-1 ] )  then
           if (0.86*@#AverageC[ Bar-1 ] > @#AverageC[ Bar ] )  then
              BuyAtMarket(Bar+1,”);
         
       if (StochK( Bar, StochPeriodShort) > 97) then
         if (1.12*@#AverageC[ Bar-2 ] < @#AverageC[ Bar-1 ] )  then
           if (1.24*@#AverageC[ Bar-1 ] < @#AverageC[ Bar ] )  then
              ShortAtMarket(Bar+1,”);
     end
     else
     {Exit rules}
     begin
       if MarketPosition = 1 then SellAtClose( Bar + 1, LastPosition, ” );
       if MarketPosition = -1 then CoverAtClose( Bar + 1, LastPosition, ” );
     end;
end;

{Plotting}
StochPane := CreatePane( 120, true, true );
PlotSeries( StochKSeries(StochPeriod), StochPane, #Purple, #Thin);
PlotSeries( StochKSeries(StochPeriodShort), StochPane, #Green, #Thin);

Popularity: 3%

Backtesting Script and Data Progress

I’ve finished multiple subroutines for calculating Simple Moving Average, Slow Stochastic, and Fast Stochastic.  I’m running the Fast Stochastic test with 2005 data from EUR/USD. 

A virtual sell order will be initiated when %k crosses below %d and a virtual buy order initiated when %k crosses above %d.  

My first test was using Slow Stochastic (5,5,5) and from 2005-01-01 to 2005-11-01, if crossover buy/sell signals were used strictly , your total loss would be 234 PIPS.

I’m going to move the stochastic parameters around and will post more results later. 

Popularity: 3%