It seems to me that the Chaldeans who are accusing the Jews are leveraging the pride of the king to get them in trouble. It was the king's decision to set it up, and it was the king who was most upset when he heard the Jews were not worshiping it.
Just look at the timeline of events. He sets up the golden image (1), he brings all the rulers to witness the dedication (2). In a sense, he is creating a state religion.
The Chaldeans also point out that it is rulers (those who were invited) that were ignoring the king's command (12).