Thanks guys
If the content of the pages are being updated then you shouldn't have to update daily / nightly as the sitemap just says "I have a page on my site and this is the name of it"
This is a uk golfing website. We have about 6 different information pages for around 320 golf courses in the UK.
Now this may be flawed in terms of seo but if a course becomes inactive on our web services it becomes inactive on our site (6 unique pages for that course are no longer available) and the user requesting the page is given a custom 404.
When that course is set to active again, the 6 unique pages for that course are available again, no more 404 for the user.
This also happens when courses are removed from our site or have just joined.
This is so sporadic, 1 course a month might leave, 2 courses a week might be set to inactive, 3 courses a week may join. Beacuse this activity is all over the place I feel as if I have to run GSiteCrawler every night. I don't want to have to generate and upload the file on an as it happens basis, i think this would be too much work (I'm a developer, not an SEO'er!)
If you have that many Pages that it will chew lots of processor time creating the sitemaps, then I suggest using multiple sitemaps - close off different sections of the site or if you have sub-sites within your sites for different languages/regions, put them into different sitemaps
Bear in mind that sitemaps are an assistant to Google's and your hosting bandwidth, so having a relly huge sitemap that takes a while to download may defeat the purpose.
I could let it run during the day, like I have done today on my first run, which took about 30 mins. You know yourself, I'm greedy with my CPU and RAM, I just don't like other applications slowing my workstation down. (Not that it was too noticeable today!) Running at night was just an idea, but in hindsight, that would be bad for the environment!
I wouldn't call the sitemap huge. It's just the time that it takes to generate the XML Sitemap and the fact that it consumes some RAM and CPU on me that are the issues! (Beggars can't be choosers!) The raw xml version of the sitemap (4000+ pages) is 858KB, the GZipped version is 41KB.
Does google use the GZipped version?
Is there an optimum file size for an XML filesize that won't banjax google and our bandwidth?