Blockbuster WSJ Investigation: How Google Interferes With Its Search Algorithms and Changes Your Results
Posted on November 17, 2019 by Jerri-Lynn Scofield
By Jerri-Lynn Scofield, who has worked as a securities lawyer and a derivatives trader. She is currently writing a book about textile artisans.
The WSJ published a comprehensive investigation Friday, How Google Interferes With Its Search Algorithms and Changes Your Results, that provides fodder for ongoing or new antitrust investigations of the company, both in the US, and worldwide:
THE JOURNAL’S FINDINGS undercut one of Google’s core defenses against global regulators worried about how it wields its immense power—that the company doesn’t exert editorial control over what it shows users. Regulators’ areas of concern include anticompetitive practices, political bias and online misinformation.
Permit to me quote from the WSJ’s takedown at length – although I encourage readers, if possible, to read the entire (paywalled) version, for it contains a wealth of information, as well as lots of cool graphics:
Google’s evolving approach marks a shift from its founding philosophy of “organizing the world’s information,” to one that is far more active in deciding how that information should appear.
More than 100 interviews and the Journal’s own testing of Google’s search results reveal:
• Google made algorithmic changes to its search results that favor big businesses over smaller ones, and in at least one case made changes on behalf of a major advertiser, eBayInc., contrary to its public position that it never takes that type of action. The company also boosts some major websites, such as Amazon.com Inc. and Facebook Inc., according to people familiar with the matter.
• Google engineers regularly make behind-the-scenes adjustments to other information the company is increasingly layering on top of its basic search results. These features include auto-complete suggestions, boxes called “knowledge panels” and “featured snippets,” and news results, which aren’t subject to the same company policies limiting what engineers can remove or change.
• Despite publicly denying doing so, Google keeps blacklists to remove certain sites or prevent others from surfacing in certain types of results. These moves are separate from those that block sites as required by U.S. or foreign law, such as those featuring child abuse or with copyright infringement, and from changes designed to demote spam sites, which attempt to game the system to appear higher in results.
• In auto-complete, the feature that predicts search terms as the user types a query, Google’s engineers have created algorithms and blacklists to weed out more-incendiary suggestions for controversial subjects, such as abortion or immigration, in effect filtering out inflammatory results on high-profile topics.
• Google employees and executives, including co-founders Larry Page and Sergey Brin, have disagreed on how much to intervene on search results and to what extent. Employees can push for revisions in specific search results, including on topics such as vaccinations and autism.
• To evaluate its search results, Google employs thousands of low-paid contractors whose purpose the company says is to assess the quality of the algorithms’ rankings. Even so, contractors said Google gave feedback to these workers to convey what it considered to be the correct ranking of results, and they revised their assessments accordingly, according to contractors interviewed by the Journal. The contractors’ collective evaluations are then used to adjust algorithms.
Biases
Big vs. Small. One major bias: a preference for big versus small. The WSJ notes that at least for shopping results, Google made the tweak as it believed consumers are more likely to find what they want at larger vendors. But this bias looks to me like it stymies, rather than promotes competition.
The bias is not limited to shopping, as WSJ reader James West noted in comments (agreeing implicitly with my interpretation of the anti-competitive effect of the Google practice):
James West
Our company, an independent publisher of financial coverage of small cap Canadian companies, has routinely been the target of what can only be explained as “manual downgrades” in Google search results. Our tests indicate a persistent pattern where Google is awarding search visibility increasingly to large US media enterprises, even where ours is local to the issue, and more detailed.
Google’s founders Page and Brin have built the company to thwart contact from its users, and now, as evidenced by WSJ’s coverage, there are widespread issues with Google’s monopoly on search. Thanks to WSJ’s coverage, we will commence a request process with Canada’s competition bureau to investigate Google for anti-competitive practices.
This company needs to be more closely regulated, as they are systematically eviscerating entire industries but reserving the bad behaviour it claims to police for its own financial gain.
Political
Conservative sites often claim their sites are disadvantaged compared to “liberal” or “mainstream” sites. This is not exactly news. Yet the bias extends beyond rightwing sites.
Yves has written about how changes to Googles’s search algorithm have whacked Naked Capitalism’s traffic (see Google Algorithm Change Whacks Naked Capitalism; Naked Capitalism is Back! Google Whackage Reversed (note the recovery in traffic was due to remedial measures Naked Capitalism undertook, rather than a Google reversal and upranking; and Google Further Crapifies Search, Exploiting Both Users and Advertisers).
As the World Socialist Web Site wrote in 2017 in Google’s new search protocol is restricting access to 13 leading socialist, progressive and anti-war web sites:
New data compiled by the World Socialist Web Site, with the assistance of other Internet-based news outlets and search technology experts, proves that a massive loss of readership observed by socialist, anti-war and progressive web sites over the past three months has been caused by a cumulative 45 percent decrease in traffic from Google searches.
The drop followed the implementation of changes in Google’s search evaluation protocols. In a statement issued on April 25, Ben Gomes, the company’s vice president for engineering, stated that Google’s update of its search engine would block access to “offensive” sites, while working to surface more “authoritative content.”
The World Socialist Web Site has obtained statistical data from SEMrush estimating the decline of traffic generated by Google searches for 13 sites with substantial readerships. The results are as follows:
* wsws.org fell by 67 percent
* alternet.org fell by 63 percent
* globalresearch.ca fell by 62 percent
* consortiumnews.com fell by 47 percent
* socialistworker.org fell by 47 percent
* mediamatters.org fell by 42 percent
* commondreams.org fell by 37 percent
* internationalviewpoint.org fell by 36 percent
* democracynow.org fell by 36 percent
* wikileaks.org fell by 30 percent
* truth-out.org fell by 25 percent
* counterpunch.org fell by 21 percent
* theintercept.com fell by 19 percent
Also on point is a Naked Capitalism crosspost of this 2018 Paul Jay interview, Matt Taibbi on Facebook and Google Playing the Censor; From the intro by Yves:
I’m glad to see Taibbi speaking out in this Real News Network interview on this issue of growing censorship by Facebook and Google and hope that more journalists join him. With the help of so many of your readers sharing our post and encouraging your friends and family members to check us out, we’ve managed to stay on an even keel, while other “deemed to be leftie” sites have taken a traffic hit due to Google downgrading non-MSM sites greatly in their search rankings. Even the Intercept, hardly a blog-scale operation, got whacked.
The problem is only getting worse. The WSJ notes that Google is increasing the aggregate number of changes to its algorithms, to about 3,200 tweaks in 2018, up from more than 2,400 in 2017, and further from about 500 in 2010.
Influence of Advertisers: Blacklisting; Paywalled Sites
The WSJ investigation discusses how Google caters to the interests of big advertisers:
Some very big advertisers received direct advice on how to improve their organic search results, a perk not available to businesses with no contacts at Google, according to people familiar with the matter.In some cases, that help included sending in search engineers to explain a problem, they said.
In another incident, the WSJ documents how Google reversed a decision that demoted the search results of certain e-Bay pages, in response to lobbying by the company, a significant advertiser.
Yves discussed the influence of advertisers blacklisting “controversial” content in this August post, Advertisers Blacklisting News, Other Stories with “Controversial” Words Like “Trump” (a piece that also keyed to a WSJ story):
It’s no longer paranoid to say that “they” are out to kill news. First it was the Internet almost entirely displacing classified ads, which had accounted for roughly half of newspaper industry revenues in the US. The Internet also turned most people save those who are now oldsters off print newspapers, even though nothing is so efficient to scan, taking with it higher subscription rates and display ads. Then Facebook and Google sucked most online advertiser revenues to themselves.
To add insult to injury, Google implemented algos hostile to smaller sites, first targeting those that did what Google deemed to be too much aggregation, like our daily Links feature. Google deemed those sites to be “low quality”. One wonders if the real issue was that they competed with Google News. Then Google downgraded sites it deemed not to be “authoritative,” whacking not only many left and right leaning sites but even The Intercept. Facebook’s parallel action was to change its search and newsfeed algos, supposedly to combat fake news, but also hurting left-leaning publishers.
Now, as the Wall Street Journal reports, many major advertisers have created blacklists, nixing ad placements that appear next to or in stories with headlines using naughty words like “bomb” that amount to a partial or total ban on news content. It isn’t isn’t just fluffy feel good brands that want to steer clear of controversy. Startlingly, even some financial services companies like Fidelity want to stay away from hot words like “Trump” even though “Trump” appears regularly in business news headlines, such as ones discussing his China trade spat, his tax cuts, his deregulatory efforts, and today, his interest in buying Greenland.
In the interests of keeping my post short, I’ve limited my quotation; I encourage interested readers to read Yves in full.
Despite maintaining in Congressional testimony that it doesn’t use blacklists, the WSJ account found that Google does. Google tries to wiggle around this apparent contradiction by relying on a narrow interpretation of what constitutes a “political” blacklist:
Google’s first blacklists date to the early 2000s, when the company made a list of spam sites that it removed from its index, one of those people said. This means the sites wouldn’t appear in search results.
Engineers known as “maintainers” are authorized to make and approve changes to blacklists. It takes at least two people to do this; one person makes the change, while a second approves it, according to the person familiar with the matter.
The Journal reviewed a draft policy document from August 2018 that outlines how Google employees should implement an anti-misinformation blacklist aimed at blocking certain publishers from appearing in Google News and other search products. The document says engineers should focus on “a publisher misrepresenting their ownership or web properties” and having “deceptive content”—that is, sites that actively aim to mislead—as opposed to those that have inaccurate content.
“The purpose of the blacklist will be to bar the sites from surfacing in any Search feature or news product sites,” the document states.
The process for creating such blacklists is opaque, so it’s difficult to determine whether there is indeed a political motivation for so doing.
And finally, the Journal discussed its own efforts to change a Google policy to disfavor outlets, such as itself, that charge for subscriptions:
(The Wall Street Journal is owned by News Corp, which has complained publicly about Google’s moves to play down news sites that charge for subscriptions. Google ended the policy after intensive lobbying by News Corp and other paywalled publishers. More recently, News Corp has called for an “algorithm review board” to oversee Google, Facebook and other tech giants. News Corp has a commercial agreement to supply news through Facebook, and Dow Jones & Co., publisher of The Wall Street Journal, has a commercial agreement to supply news through Apple services. Google’s Ms. Levin and News Corp declined to comment.)
The Bottom Line
To sum it all up: Thomas Ferguson notes in an email “the last third of the WSJ article basically describes ‘electronic corporatism’ in which big private tech concerns look out for each other. Everyone else can’t even get an answer.”
What Is to be Done?
Some if not much of this info is well known to at least some antitrust regulators. Yet by publishing it, the WSJ increases pressure on them to address the problems Google’s dominance raises. The company currently captures more than 90% of the market share for all search engines.
As to those ongoing antitrust investigations, Matt Stoller, writing in the Guardian a few months ago, The great breakup of big tech is finally beginning, summarized the then-state of play:
Last week, state attorneys general, led by Texas and New York, announced investigations into Google and Facebook for possible antitrust violations. This is a big deal. No society has ever centralized control of information as we have in big tech, and this is the first real American strike at the problem. As Scott Galloway frequently notes in his podcast with tech journalist Kara Swisher, the big tech breakup has finally begun.
Note that this is one of many areas where Trump inattention or inaction doesn’t really matter. The feds aren’t the only game in town, and attorney general from US states, as well as the EU and many other countries, are on the case – not to mention Congress critters. Over to Stoller:
Normally, antitrust enforcement would come from the federal government, but Trump enforcers have proved irrelevant at best. Instead these investigations are being led by the states. The Republican attorney general of Texas and the Democratic attorney general of New York are informal leaders, meaning that the investigations are bipartisan. The state attorneys general complement an important investigation by the House antitrust subcommittee led by David Cicilline. Such leadership suggests the rule of law, absent from American business for several decades, may be on its way back. There are also important investigations, hearings or cases by enforcers in Germany, France, the European Union, Israel, India, Singapore, Russia, Mexico and Australia, among others.
(I should mention Stoller here has a new book out, Goliath: The 100-Year War between Monopoly Power and Democracy,which I’ve purchased (from my friendly independent bookstore). It’s next up in my to-read queue after I finish William Dalrymple’s The Anarchy, The Relentless Rise of the East India Company. I got sidetracked and polished off Matt Taibbi’s Hate, Inc. after seeing John Siman’s rave review, Manufacturing Fear and Loathing, Maximizing Corporate Profits! A Review of Matt Taibbi’s Hate Inc.: Why Today’s Media Makes Us Despise One Another).
From WSJ comments:
Steve W. Bell
This is a terrific piece of Journalistic work – just when I thought the WSJ no longer was capable of it, comes this which is a Pulitzer class expose’. It is fair and objective., and introduces strong evidence.
A 4th Grader could easily discern that Google search results are sharply biased and shaped to suppress legitimate Conservative speech that Google doesn’t like (that is, most all of it) from search results.
Google took a dark turn, in my view, 2-3 years ago. I do not believe that their many great mid-level employees are the reason. They changed out the Sr. management team 3 years back. Now, two immense anti-trust investigations underway.
Google has also turned sharply arrogant, e.g. on Jan 1st shifting all support calls – even from Agencies, to barely-trained staff in India.
Breaking up Google into 2 or 3 regulated entities would be great for Google, it’s employees AND consumers. Although they enjoy a dominant market position, Google is a commodity – EASILY replaced.
To repeat what I said above, the political bias extends beyond downranking conservative sites, to obscuring the output of other sites that also may have something to say.
