As an example, the project I work on uses an rc file for this:
https://raw.githubusercontent.com/raynebc/editor-on-fire/master/src/eof.rc
https://raw.githubusercontent.com/raynebc/editor-on-fire/master/src/eof.ico
The RC file is a simple text file with this content:
allegro_icon ICON myfilename.ico
That's with a carriage return and line feed at the end of the line (Windows style encoding), in case this makes a difference. The ICO file itself seems to be a pretty standard Windows icon image. When I open that in GIMP, it appears to have 5 layers, each with the graphic in a different size.