Screen Scraping

[Mitch]

I'm not sure whether or not it's right to post a question like this on this forum, but you're all interested in the same things ie. making money on Betfair and there's obviously some very clever people use this so I'll give it a go.

If I get slated for asking then I aopologise and I'll leave it.

Gary's application has got me so interested in the possibilities open to us on Betfair that I am trying to use screen scraping to access the Betfair site myself.
I have no previous experience of doing this but have managed through trial and error to get a list of markets, then individual market info (in seperate progs) with the help of an http analyzer.

That's the easy part because it's just receiving the info, but I'm struggling when I want to send info, like logging in etc.

Can anyone point me in the right direction for finding out what the Betfair site wants to see, and how it should be presented?

Thanks for reading.

[GeorgeUK]

Not really much help here Mitch, but what you could look at are excel to web forms.

This is often associated with questionairre type stuff, but also deal with site login etc.
When i looked at the betfair website, to get the matkets, it was a table within a frame, within a frame, within a frame. I can get the tables and tablerows (TR) and then the individual cells into excel - but haven't tried the login part yet either. One thing i would say - do not use sendkeys. Way too volatile.

I don't have access to a computer i can load a "Document Object Model" tool on. I would imagine you would need to navigate to the appropriate table, then perhap getElementById to find the appropriate section that will require you to onput the username and password.

Sorry it's so vague - like i said, don't have access to anything i can look at right now. But i think this gives the general idea

with Document
.getElementById("txtLogin").Value = "username"
.getElementById("txtpassword").Value = "password"
.getElementById("submit1").Click

[Mitch]

Thanks for the reply George.

I'm not using Excel for this one, I'm having a go with Java

Also, I'm not trying to manipulate the BF site through IE, but rather send requests to the server. Like I said, I've managed to get market info but I might have gone about it completely the wrong way

[a1ehouse]

Try here Mitch: http://www.punterslounge.com/forum/foru ... y.php?f=26 all the stickies

[Captain Sensible]

Mitch I don't use java so no idea of the web commands but the login screen is just

https://www.betfair.com/account/login/L ... urpassword

that'll return

http://www.betfair.com/account/login/Lo ... us=success

and all your cookie info that needs to be sent when placing a bet

No need to login if you're just scraping, only when placing bets. The betplacement request for the web interface is long

http://www.betfair.com/betting/BetsMani ... &BetType=O

any value starting $ is one I enter.

If you don't do loads of bets a minute I'd consider the freeapi as it's easier to use and if you need to scrape alot you can always screen scrape and then place bets via API

[Mitch]

Thanks both.

a1ehouse - that is the site that I've got what little knowledge I have from, it's well worth checking out for anyone else interested in this sort of thing.

Captain - That's exactly the sort of thing I'm looking for but the stream I used to capture the reply showed nothing! Is that because it's an HTTPS address or will I have done something wrong?

How did you learn? Through messing about or a website or something else? Do you use an http analyzer that you'd recommend - the one I'm using now is quite complicated..... all I want to see is what's sent and what comes back I think.
What do you use for sending and receiving?


Thanks again.

[Captain Sensible]

Probably because it's an https address. You have to look up the java functions for connecting to a secure server (SSL). I used php when scraping cos it was free and I had some experience with html but connecting to a secure server with php was a pain. I now use php5 and the free api as php has all the https libraries written in. If I started now I'd probably look at another language than php though.

I'd check out the api because evrything is structured for you - no idea how clued up with java you are but there's free api code at

http://bdpsupport.betfair.com/dmfiles/2 ... le4_1.java

To find the info for scraping I used a cracked version of IEWatch which isn't complicated to read (if you want a copy ?) some of the analyzers I used prior to hat gave too much info that wasn't needed.

I used php to send and receive because it runs in a web browser just like the betfair interface just meant I could scrape and display the bits I needed rather than the other stuff like graphics plus you can scrape things like sporting life bettings odds at the same time and have them all displayed on the same page comparing bf odds against sp etc.

I do think the free api should be your first call as it's all structured for you and you wont have to write your own routines - if you find the bet placement limits oo restricive you can then write your own routines.

http://bdphelp.betfair.com/API4/V1/API4 ... wwhelp.htm

[Mitch]

How clued up am I? Hmmmmmmmm - not at all.

I started reading a Java book a couple of months ago (starting with no idea whatsoever) and in the last few days got bored and frustrated with it because I don't really like learning like that. I'd much rather learn by trying something I'm interested in so I modified the code at the punterslounge (pointed out above) to suit my own ends and it works for basic things as I mentioned at the start.

The sample code on the bdp site looks like it needs about 70 pre-written classes importing and I don't have them. I guess they must be available from somewhere but my searches haven't found them, but I think I'd be best sticking with simpler things for now as it's getting very confusing.

I'll try iewatch, I see they have a 21-day trial available. Maybe that will be long enough to determine which parts are the important things in the traffic.

The most frustrating thing is that I know (maybe delusional?) that I'm not a dummy but this is very slow going indeed.

Thanks again for all the help.

[PeteB]

Mitch,

I'd also recommend a look at the free API - I am working on a C# .NET program (teaching myself C# as I go - with http://tinyurl.com/o4g9o), and it's all been very straightforward so far.
I got of a shock when I looked at the sample excel program on the bdp forum, as it has a humungous amount of potentially buggy code.
With C#, there is a program called wsdl.exe which you point at the API URL, and it automatically generates all the code needed to interact with each API function, and then all the XML handling is built into C# .NET. So I only had to write a tiny amount of code to pop up a dialog to let you select a market, and then to get the market and selection data, including the available depth and traded volume as per the popup window with graph on the website, and getting matched and unmatched bets. I haven't done placing bets yet.

C# is a pretty easy step from java, especially if you are only starting with java. If you're interested, I can send you what I've done so far, and you'll be able to see how little code you have to write. (In a few weeks after hols I should have a solution that puts all the API data into Excel - I should be able to share this more widely at that stage, but not yet...)

Overall, if the API has what you need, then you will need to write (and test) much less code this way than java and screenscraping, and will be insulated from website changes.

Pete.

[PeteB]

Re bet placement limits with FreeAPI:

The table of the throttling times is here: http://bdp.betfair.com/freeapi_table.php

Everything you need to do regularly you can do 60 times a minute - I think this is enought for trading a single market or selection at a time. You can place / update / cancel multiple bets in a single call, so this should be ok. It means building a program that checks the market and acts on it every second.

(at this stage I can only dream of wirting something that works well enough to pay for the API so it can trade multiple markets at once!)

[Mitch]

Thanks for the info, I'll take a look.