How the Twitter App Bypasses Paywalls

by Isoroku Yamamoto

Wall Street Journal ended its practice of allowing special access for search engines. This means that a human visitor can no longer bypass the paywall by spoofing Google’s HTTP request headers.

However, subscription-based publications face a problem when users click on a link through Twitter or Facebook on a mobile device. Social media apps implement their own in-app browser, which generally do not retain cookies. Websites that require a user login must request the login every time the app is reopened.

This makes for a cumbersome user experience. Thus, publications like the Wall Street Journal disable login checks when a page request appears to come from Twitter.

It does this by inspecting HTTP request headers. The important headers are Referer and User-Agent.

When a link is shared on Twitter, the url is shortened to something like “https://t.co/9Mk58nL3xJ.” This goes to a Twitter server, which redirects the browser to the intended destination. Websites determine whether Twitter initiated the redirect by checking that the HTTP Referer string begins with “https://t.co/.” The rest of the string is ignored.

A web request from Twitter further identifies itself through the User-Agent header, which might look something like “Mobile/14C92 Twitter for iPhone.”

By submitting this information in request headers, any web browser can appear to be the Twitter app. It is easy to do this using a Chrome extension.

The following builds on top of last year’s tutorial for mimicking Google’s web crawler.

1. Use the same manifest.json file as before. Take care to list both http:// and https:// versions of the sites you are interested in, as many publishers now use ssl.

2. Modify the background.js file. The modified version should look like the one below. It is worth noting that all cookies have been blocked.

var VIA_TWITTER = ["wsj.com"]

function changeRefer(details) {

  foundReferer = false;
  foundUA = false;

  var useTwitter = VIA_TWITTER.map(function(url) {
    if (details.url.includes(url)) {
      return true;
    }
    return false;
  })
  .reduce(function(a, b) { return a || b}, false);

  var reqHeaders = details.requestHeaders.filter(function(header) {

    // block cookies by default
    if (header.name !== "Cookie") {
      return header;
    } 

  }).map(function(header) {
    
    if (header.name === "Referer") {
      header.value = setRefer(useTwitter);
      foundReferer = true;
    }
    if (header.name === "User-Agent") {
      header.value = setUserAgent(useTwitter);
      foundUA = true;
    }
    return header;
  })
  
  // append referer
  if (!foundReferer) {
    reqHeaders.push({
      "name": "Referer",
      "value": setRefer(useTwitter)
    })
  }
  if (!foundUA) {
    reqHeaders.push({
      "name": "User-Agent",
      "value": setUserAgent(useTwitter)
    })
  }
  return {requestHeaders: reqHeaders};
}

function blockCookies(details) {
  for (var i = 0; i < details.responseHeaders.length; ++i) {
    if (details.responseHeaders[i].name === "Set-Cookie") {
      details.responseHeaders.splice(i, 1);
    }
  }
  return {responseHeaders: details.responseHeaders};
}

function setRefer(useTwitter) {
  if (useTwitter) return "https://t.co/T1323aaaa"; 
  else return "https://www.google.com/";
}

function setUserAgent(useTwitter) {
  if (useTwitter) return "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.1.32 (KHTML, like Gecko) Mobile/14C92 Twitter for iPhone";
  else return "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";
}

chrome.webRequest.onBeforeSendHeaders.addListener(changeRefer, {
  urls: ["<all_urls>"],
  types: ["main_frame"],
}, ["requestHeaders", "blocking"]);

chrome.webRequest.onHeadersReceived.addListener(blockCookies, {
  urls: ["<all_urls>"],
  types: ["main_frame"],
}, ["responseHeaders", "blocking"]);

Save both files in the same directory. The updated source code can also be downloaded here.

Now type chrome://extensions/ in the browser address bar.

Reload the old extension, or Load it as an unpacked extension if you have not previously done so. Enable the chrome extension and visit wsj.com.

There is always a tradeoff between security and usability. The fastest way to compromise a computer system is to accommodate lazy users. Or worse yet, accommodate lazy programmers.

How Google’s Web Crawler Bypasses Paywalls

by Isoroku Yamamoto

Update: A newer version of the chrome extension is available here.

Wall Street Journal fixed their “paste a headline into Google News” paywall trick. However, Google can still index the content.

Digital publications allow discriminatory access for search engines by inspecting HTTP request headers. The two relevant headers are Referer and User-Agent.

Referer identifies the address of the web page that linked to the resource. Previously, when you clicked a link through Google search, the Referer would say https://www.google.com/. This is no longer enough.

More recently, websites started checking for User-Agent, a string that identifies the browser or app that made the request. Wall Street Journal wants to know that you not only came from Google, but also that you are an agent of Google.

By providing this information in request headers, anyone can appear to be a Google web crawler. In fact, I will show you how to make a Chrome extension that does just that.

1. Create a file called manifest.json. Paste the following in the file. Add any sites you would like to read to the permissions list.

{
  "name": "Innocuous Chrome Extension",
  "version": "0.1",
  "description": "This is an innocuous chrome extension.",
  "permissions": ["webRequest", "webRequestBlocking",
                  "http://www.ft.com/*",
                  "http://www.wsj.com/*",
                  "https://www.wsj.com/*",
                  "http://www.economist.com/*",
                  "http://www.nytimes.com/*",
                  "https://hbr.org/*",
                  "http://www.newyorker.com/*",
                  "http://www.forbes.com/*",
                  "http://online.barrons.com/*",
                  "http://www.barrons.com/*",
                  "http://www.investingdaily.com/*",
                  "http://realmoney.thestreet.com/*",
                  "http://www.washingtonpost.com/*"
                  ],
  "background": {
    "scripts": ["background.js"]
  },
  "manifest_version": 2
}

2. Create a file called background.js. Paste the following into the file:

var ALLOW_COOKIES = ["nytimes", "ft.com"]

function changeRefer(details) {
  foundReferer = false;
  foundUA = false

  var reqHeaders = details.requestHeaders.filter(function(header) {
    // block cookies by default
    if (header.name !== "Cookie") {
      return header;
    } 

    allowHeader = ALLOW_COOKIES.map(function(url) {
      if (details.url.includes(url)) {
        return true;
      }
      return false;
    });
    if (allowHeader.reduce(function(a, b) { return a || b}, false)) { return header; }

  }).map(function(header) {
    
    if (header.name === "Referer") {
      header.value = "https://www.google.com/";
      foundReferer = true;
    }
    if (header.name === "User-Agent") {
      header.value = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";
      foundUA = true;
    }
    return header;
  })
  
  // append referer
  if (!foundReferer) {
    reqHeaders.push({
      "name": "Referer",
      "value": "https://www.google.com/"
    })
  }
  if (!foundUA) {
    reqHeaders.push({
      "name": "User-Agent",
      "value": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    })
  }
  console.log(reqHeaders);
  return {requestHeaders: reqHeaders};
}

function blockCookies(details) {
  for (var i = 0; i < details.responseHeaders.length; ++i) {
    if (details.responseHeaders[i].name === "Set-Cookie") {
      details.responseHeaders.splice(i, 1);
    }
  }
  return {responseHeaders: details.responseHeaders};
}

chrome.webRequest.onBeforeSendHeaders.addListener(changeRefer, {
  urls: ["<all_urls>"],
  types: ["main_frame"],
}, ["requestHeaders", "blocking"]);

chrome.webRequest.onHeadersReceived.addListener(blockCookies, {
  urls: ["<all_urls>"],
  types: ["main_frame"],
}, ["responseHeaders", "blocking"]);

Save both files in one directory. These should be the only files in the directory. If you were too lazy to copy and paste, you can download the source code here.

Now type chrome://extensions/ in the browser address bar.

Click Load unpacked extension... (Make sure Developer Mode is checked in the upper right if you do not see the buttons.)

Screen Shot 2016-02-18 at 10.49.25 PM

Select the directory where you saved the two files. Enable the chrome extension and visit wsj.com.

Remember: Any time you introduce an access point for a trusted third party, you inevitably end up allowing access to anybody.

I am Fond of Pigs

by Winston S. Churchill

Dogs look up to us. Cats look down on us. Pigs treat us as equals.

pig

In America we succeeded in eliminating hereditary power, in obtaining a large measure of political liberty, only to see the rise of an economic power, and the consequent loss of economic liberty. The industrial development of the United States was of course a necessary and desirable thing, but the economic doctrine which formed the basis of American institutions proved to be unsuited to industrialism, and introduced unforeseen evils that were a serious menace to the Republic. An individualistic economic philosophy worked admirably while there was ample land for the pioneer, equality of opportunity to satisfy the individual initiative of the enterprising. But what is known as industrialism brought in its train fear and favour, privilege and poverty, slums, disease, and municipal vice, fostered a too rapid immigration, established in America a tenant system alien to our traditions.

Before the advent of industrialism, poverty was negligible, there was no great contrast between rich and poor; the artisan, the farmer, the well-to-do merchant met on terms of mutual self-respect, as man to man; economic class consciousness was non-existent; education was so widespread that European travellers wonderingly commented on the fact that we had no “peasantry”; and with few exceptions every citizen owned a piece of land and a home.

Property, a refuge a man may call his own, and on which he may express his individuality, is essential to happiness and self-respect. Today, less than two thirds of our farmers own their land, while vast numbers of our working men and women possess nothing but the labour of their hands. The designation of labour as “property” by our courts only served to tighten the bonds, by obstructing for a time the movement to decrease the tedious and debilitating hours of contact of the human organism with the machine,—a menace to the future of the race, especially in the case of women and children. If labour is “property,” wretches driven by economic necessity have indeed only the choice of a change of masters. In addition to the manual workers, an army of clerical workers of both sexes likewise became tenants, and dependents who knew not the satisfaction of a real home.

Such conditions gradually brought about a profound discontent, a grouping of classes. Among the comparatively prosperous there was set up a social competition in luxury that was the bane of large and small communities. Skilled labour banded itself into unions, employers organized to oppose them, and the result was a class conflict never contemplated by the founders of the Republic, repugnant to democracy which by its very nature depends for its existence on the elimination of classes. In addition to this, owing to the unprecedented immigration of ignorant Europeans to supply the labour demand, we acquired a sinister proletariat of unskilled economic slaves. Before the war labour discovered its strength; since the war began, especially in the allied nations with quasi-democratic institutions, it is aware of its power to exert a leverage capable of paralyzing industry for a period sufficient to destroy the chances of victory. The probability of the occurrence of such a calamity depends wholly on whether or not the workman can be convinced that it is his war, for he will not exert himself to perpetuate a social order in which he has lost faith, even though he now obtains a considerable increase in wages. Agreements entered into with the government by union leaders will not hold him if at any time he fails to be satisfied that the present world conflict will not result in a greater social justice. This fact has been demonstrated by what is known as the “shop steward” movement in England, where the workers repudiated the leaders’ agreements and everywhere organized local strikes. And in America, the unskilled workers are largely outside of the unions.

The workman has a natural and laudable desire to share more fully in the good things of life. And it is coming to be recognized that material prosperity, up to a certain point, is the foundation of mental and spiritual welfare: clean and comfortable surroundings, beauty, rational amusements, opportunity for a rational satisfaction of, the human. instincts are essential to contentment and progress. The individual, of course, must be enlightened; and local labour unions, recognizing this, are spending considerable sums all over the country on schools to educate their members. If a workman is a profiteer, he is more to be excused than the business profiteer, against whom his anger is directed; if he is a spendthrift, prodigality is a natural consequence of rapid acquisition. We have been a nation of spendthrifts.

A failure to grasp the psychology of the worker involves disastrous consequences. A discussion as to whether or not his attitude is unpatriotic and selfish is futile. No more profound mistake could be made than to attribute to any element of the population motives wholly base. Human nature is neither all black nor all white, yet is capable of supreme sacrifices when adequately appealed to. What we must get into our minds is the fact that a social order that insured a large measure of democracy in the early days of the Republic is inadequate to meet modern industrial conditions. Higher wages, material prosperity alone will not suffice to satisfy aspirations for a fuller self-realization, once the method by which these aspirations can be gained is glimpsed. For it cannot be too often repeated that the unquenchable conflicts are those waged for ideas and not dollars.