Can I install a Shelly in the roller shutter box?

You will most likely need to pull additional cables from the switch box to the roller shutter box. It might be easier to enlarge the switch box e.g. by drilling it deeper. But you would need to provide more details about what kind of switch box it is.