How to use setprop via adb shell in recovery?

Firstly the main Android OS (what you call "boot mode") and recovery are two independent operating systems. When one is running, other is not. Running processes are different, environment is different, loaded properties are different and so on.

Recovery has its own default.prop file in ramdisk from which it reads properties. In order to permanently change the value of ro.product.device you'll have to unpack recovery.img (or boot.img on A/B devices), modify the file, repack and re-flash.

Secondly you can't set read-only (ro.*) properties once they are already set. Magisk's resetprop tool however can do this, going a little out of the way.

Thirdly ro.product.device is usually set in /system/build.prop. So you can edit this file and reboot device to see changes. There are also some other files from which properties are set on boot.

^{Make sure not to break dm-verity by modifying /system if enabled.}

Since the (/system and other) partitions are accessible from recovery mode too, you can mount filesystem read/write in recovery and edit the file.

Fourthly the value you are getting in fastboot is independent of both main OS and recovery. That's hard-coded in bootloader. You'll get that value even if there is no recovery and no Android OS on device.