Google’s attempts to settle its legal dispute with authors and publishers — and in the process, claim the rights to millions of out-of-print titles — have been dismissed by a federal judge.

Judge Denny Chin said the proposed settlement would go too far in granting rights to Google and even said that if approved, it would be “rewarding [Google] for engaging in wholesale copying of copyrighted works without permission.”

The case involves a lawsuit by the Authors Guild and the Association of American Publishers dating back to 2005, which accused Google of copyright infringement by scanning books to make them searchable online.

In 2008 the two sides agreed a settlement to the case that would include Google setting up and maintaining a register of book copyright and a royalty payment system. The settlement also gave Google the right to scan all books unless and until the copyright holder asked it to stop. And most controversially, the deal meant Google would have exclusive rights to digitize orphan titles: those which were in copyright, but where the rights holder either wasn’t known or couldn’t be traced.

Critics of the deal complained it would effectively give Google a monopoly in the book digitization market. There were also claims that neither party in the agreement had any authority to “give away” the rights to orphan books, along with serious questions about how the deal could affect books published outside the US.

According to Chin’s ruling, around 500 people filed formal responses to the proposed settlement, the vast majority opposing it. Specific objections included other author and publisher groups noting that they should be involved in any granting of rights to Google, and concerns that approving the deal would violate the right of Congress to decide copyright law.

Chin backed most of these concerns, as well as noting that given the original lawsuit only involved Google displaying “snippets” of book content on its Book Search site, a settlement wasn’t the place for the two sides to decide copyright issues involving the full text of books.

Concluding that the proposed settlement was “not fair, adequate and reasonable”, Chin rejected it. While technically that leaves the two sides back in opposition to one another with a copyright infringement claim to dispute, Chin did strongly hint that a revised deal would stand a better chance of getting approval with one major revision: that the opt-out system be replaced with opt-in, meaning Google could only digitize a book with the express permission of the rightsholder.