Building a comparison website

[betman]

I hope this is the correct forum.

Can anyone give me the basics behind building a site that compares dynamic content such as betting odds, insurance quotes, flight prices etc. Do you need to get access to content from the site owner or is there other methods e.g. scraping, crawling?

Thanks,
-betman

 

[mneylon]

A lot of the bigger sites make their data available via xml, csv etc.,

 

[googlebomber76]

good question, I'm thinking of getting a site going which works similar to callcosts.ie, is there much to it once you have all the data, are there any ready made templates that will give you this functionality?

 

[vijay]

I think there are many tutorials sites in the search engine like google and yahoo i think these sites can really help u a lot in ur task.

 

[mneylon]

I think there are many tutorials sites in the search engine like google and yahoo i think these sites can really help u a lot in ur task.

Are you just randomly posting replies or do you actually read the threads you're replying to?

 

[Gala]

I reckon screen scraping would be a good topic to explore if your planning on building this type of site. Ruby and Perl are pretty damn good at it.

There are a lot of different ways you could do this and it depends on just how dynamic the data that your capturing is. e.g: betting odds - short lived and change all the time so there is no point in archiving the data.. every time someone accesses the site the odds will have to be scraped from another site. so:

you need to find out the structure of the sites your scraping ie: where abouts in the html docs your scraping are the important data items held? Once you know what your scraping you can then build your code. your going to need to send a http request to their server, pull back the html, maybe coverrt it to xml (e.g: ruby RXML) and then use something like XPath to pull the nodes and values that you require into your web front end.

This could be done with a post your server or with an AJAX call. It's likely that your going to see a few performance issues around this though, so be aware of it.

This is a very high level opinion of how to do this task though and theres a milion and one ways you could do it