A penalty by Sehrou Guirassy, after the French striker had been fouled by Mathieu Debuchy, earned Cologne only their second win in 17 Bundesliga and Europa League games this season.
The closest the Gunners came to scoring was when Francis Coquelin hit the post when the game was goalless in the first half.
Arsenal have 10 points from five games, four ahead of Cologne and Red Star Belgrade, who were held to a goalless draw by Bate Borisov.
More German misery for the Gunners
Of the five defeats Arsene Wenger’s side have endured this season, this was the least damaging after qualifying for the last-16 stage with two games to spare.
Yet a much-changed Gunners line-up failed to build on the afterglow of victory over neighbours Tottenham on Saturday as they suffered another loss in Germany.
They were hammered 5-1 by Bayern Munich in the Champions League in February 2017 and November 2015, as well as losing 2-0 to Borussia Dortmund in September 2014.
Wenger changed the entire starting XI from the north London derby, forward Danny Welbeck returning from a groin injury after missing the past seven games.
Arsenal’s side, which featured Olivier Giroud and Jack Wilshere, had only 16 Premier League starts between them this season and it was a lacklustre performance against the Bundesliga’s bottom club.
Coquelin twice went close to scoring his first goal for four years, the French midfielder fizzing a shot narrowly wide from 20 yards before another effort bounced off the foot of the Cologne post.
Wenger’s side increased the tempo in the closing stages – 17-year-old substitute Reiss Nelson dancing his way around home defenders before forcing a save, while Wilshere also went close from outside the area.
Yet this a disappointing performance from the Gunners, who will look to end the group stage with a home win over Bate Borisov on 7 December.