Google warning: is your site abused through redirects?

Google recently wrote in one of its official blogs that it is possible for spammers to take advantage of your website without ever setting a virtual foot in your server. Spammers can do this by abusing open redirects.

What are open redirects?

Many websites use links that redirect their website visitors to another page. Some redirects are left open to any arbitrary destination. These redirects can be abused by spammers to trick web surfers and search engines into following links that seem to be pointing to your website although they redirect to a spammy website.

That means that people who think that they visit your website will be redirected to highly questionable web pages that might contain adult content, viruses, malware or phishing attempts.

Which redirects on your website could be abused?

Spammers are very inventive. According to Google, they have managed to use the redirect spam on a wide range of websites, including the websites of large well-known companies and the websites of small local government agencies.

For example, the following redirection types can be abused:

1. Scripts that redirect users to a file on the server can be abused by spammers. The links on your website could look like this:

   http://www.example.com/download.php?url=http://www…

   http:///www.example.com/get/pdf/?http://www…

2. Site search result pages with automatic redirect options. If the result pages of your internal site search feature contain an URL variable that sends your website visitors to other pages, spammers might be able to exploit them:

   http://www.example.com/search?q=keyword&page=1&url=…

3. Affiliate tracking links. Affiliate tracking links often allow people to direct website visitors to other pages. Spammers might enter their own URLs in the tracking links. Example:

   http://www.example.com/track.php?affid=123&url=…
   

4. Proxy pages. Proxy sites send people through to other websites and they can be abused by spammers:

   http://myproxy.example.com/?url…
   

5. Interstitial pages. Some websites show an interstitial page when users leave a website to let users know that the information found on the link is not under their control. These URLs usually look like this:

   http://www.example.com/redirect/http://www…

   http://www.example.com/out?http://www…

   http://www.example.com/cgi-bin/redirect.cgi?http://www…

How to find out if your website is abused

Even if you find none of the URLs above on your website, your site still may have open redirects. Do the following to check if your website is abused by spammers:

1. Make a site search on Google

   Go to Google.com and search for “site:yourdomain.com”. Replace yourdomain.com with your own domain name. If you see web pages that have nothing to do with your website then it’s likely that someone exploits a security hole on your website.

2. Check your web server logs for URL parameters like “=http:” or “=//”. If your redirection URLs get a lot of traffic, this could also be caused by spammers.
3. If you get user complaints about content or malware that you know cannot be found on your website then your website users might have seen your URL before they were redirected to the malware site.

What you can do to protect your website

It’s not easy to to make sure that your redirects aren’t exploited. The reason for that is that an open redirect is not a bug or a security flaw. There are some things that you can do to protect your website:

1. Check the referrer. Your redirect scripts should only work if they area accessed from another web page of your website. The redirect script should not work if the user accesses the script directly or from a search engine.
2. If possible, make sure that the script can only redirect to web pages and files that are on your own websites. You could use a whitelist of allowed destination domains.
3. Use the robots.txt file of your website to exclude search engines from the redirect scripts on your website. That will make your website less attractive for hackers.
4. Add a signature or a checksum to your redirect links so that only you can use the script.

Open redirect abuse is a big issue for Google right now. If you secure your scripts, spammers will move over to other websites and leave your website alone.

Do search engines think that your website is spam?

About three weeks ago, Microsoft was granted a new patent with the name Web Spam Classification Using Query Dependent Data. Although this patent application was filed by Microsoft, all major search engines probably use similar methods to classify web pages.

How do search engines analyze web pages?

Search engines look at a number of elements that can appear on web pages and within queries that web surfers use to find these pages.

For example, search engines may look for the most frequent keyword in the web page, the number of times a particular keyword appears in the web page, the domain name associated with the web page, the number of links pointing to the page, the HTML tags in which a keyword appears and many other factors.

The patent filing indicates that search engines look at hundreds of different factors to rank web pages.

How search engines try to detect spammy pages

The are so many potential spam pages on the Internet that search engines cannot identify all spam pages manually.

To identify potential spam pages, search engines might manually label some web pages as spam and then take information from that pages to find other spam pages.

For example, a web page that uses keyword stuffing has more keywords than a legitimate page. By training the spam detection algorithm with a few web pages that use keyword stuffing, other web pages that use keyword stuffing can be detected automatically.

In other words, a spam detection algorithm labels web pages as spam or not spam by looking at decisions made by humans. According to the patent application, the algorithm might look at the following factors:

• the number of inbound links coming from labeled spam pages
• the top level domain of the site
• the quality of phrases in the document and density of keywords (spammy terms)
• the count of the most frequent term
• the count of the number of unique terms
• the total number of terms and the number of words in the path
• the number of words in the title
• the rank of the domain and the average number of words
• the top-level domain
• the number of hits within a domain
• the number of users of a domain
• the number of hits on a URL and the number of users of a URL
• the date the URL was crawled, the last date page changed
• many more factors

If your website uses similar elements as the spammy web page then it’s likely that your website will be classified as spam. The usual impact of a website being labeled as spam is that the site might be pushed down in search results, or removed completely.

What does this mean for your website?

You should make sure that your web pages use similar elements as the top ranked pages instead of elements that can be found on spam pages.

SEO Link Building

Five mistakes that keep search engine robots away from your website

Many webmasters don’t get high rankings on Google and other search engines just because Google’s indexing robot has difficulty to index their web pages.

Search engine robots are very simple software programs. If an indexing robot cannot find the content of your website immediately, it will skip your site and go to the next link in the list. For that reason, it is very important to make sure that search engine robots can index your web pages without problems.

Here are the top 5 elements that drive search engine robots away:

Reason 1: Your robots.txt file is damaged or it contains a typo

If search engine robots misinterpret your robots.txt file, they might completely ignore your web pages.

Double check your robots.txt file and make sure that you use the disallow parameter only for web pages that you really don’t want to have indexed.

Reason 2: Your URLs contain too many variables

URLs with many variables can cause problems with search engine robots. If your URLs contain too many variables, search engine robots might ignore your pages.

Here’s Google’s official statement about web pages with many variables:

“Google indexes dynamically generated webpages, including .asp pages, .php pages, and pages with question marks in their URLs. However, these pages can cause problems for our crawler and may be ignored.”

Reason 3: You use session IDs in your URLs

Many search engines don’t index URLs that contain session IDs because they can lead to duplicate content problems. If possible, avoid session IDs in your URLs. Better use cookies to store session IDs.

Reason 4: Your web pages contain too much code

Of course, your web pages can contain JavaScript code, CSS code and other script code that is not directly related to your content. Visit your website with a web browser and select “View source” or “View HTML source”.

If it is difficult for you to spot the actual content of your website then search engines might also have difficulty to parse your pages.

Reason 5: Your website navigation causes problems

Fancy JavaScript or DHTML menus cannot be parsed by most search engine robots. Flash or AJAX menus are even worse when it comes to website navigation.

As mentioned above, search engine robots are very simple programs. They can follow HTML links, all other links can cause problems.

Optimized web page content and good inbound links are crucial for high search engine rankings. However, the best content and the best links won’t help you much if search engines cannot index your pages.

Make sure that search engine spiders can index your web pages without problems so that your web pages can get the rankings they deserve.