Just ran into the same issue today on Manjaro. I could get around the missing libavcodec.so.54, libavformat.so.54 and libavutil.so.52 by installing some compat lib versions from the AUR. However the missing libswcale.so.2 had no way to fix.
However it runs perfectly fine with the rbdoom-3-bfg 1.1.0preview3.r122.g0e2890f9-1 package from the AUR. Maybe updating the rbdoom-3-bfg package in Lutris can fix this.