. . .blame it on the next guy! Some of you have been asking us about the first LinkInTxt crawl and we have good news and bad news. Which do you want first? Ok, ok. I’ll give you the good news first. The good news is we were just a few hours away from a successful complete crawl with literally thousands upon thousands of keywords ready to be bought up. During a test, we found the most unique keywords one could imagine, meaning it will work for almost any niche website out there.
Now I guess the time has come to announce the bad news. Did you hear that business.com is for sale to whoever has a spare FOUR HUNDRED MILLION in their pocket? Oh right, the bad news. 
Unfortunately, there was a miscommunication with our server administrators regarding a routine hard drive corruption test that required a restart of our operating system. This restart occurred this morning about three hours before the entire process was going to end. We expected there was a way to simply resume the process, however, since it was ended abruptly, it corrupted all of our data. Can you say “BUMMER!”
Where does this leave us? “On the road again . . .” crawling again. Smarter. Wiser. Faster. If we want to go tech-talk, we might call this LinkInTxt v2.1.
6 comments ↓
Oh, damn!… That sucks. So, being that you started the crawl 10 days ago, is it going to take another 10 days for it to be complete again?
Will publishers be able to see crawl stats, etc on their own sites?
Yes, it took right at 9 days to crawl our first group of hundreds of large sites. We kind of went big the first go ’round to test our limitations and spot any weak areas. We definitely learned what “not” to do the hard way, but this next go round should be quicker. We made some tweaks that should cut down on a lot of time and space.
Ok, I feel I’m talking in circles. I need bed.
That has to be one of the most frustrating things you guys have had to endure. Talk about bugs in the system.
I can’t believe I haven’t seen a scathing response from Frio yet!
ODLS … yes, I think it ranks up there with the top blunders of our company’s life. But we’re going to look at it as a learning process (serenity now, serenity now..). We were able to spot resource hog directories that we can eliminate and weed out crawling things we don’t need to crawl.
I cant wait for the next crawl
Leave a Comment