Why allow permission button is disabled?

Problem

Today I noticed strange behavior on my test phone, which is Nexus 5 with Android 6.0.1.

After opening test app that asks for permission to read contacts allow button is not working - after tapping on allow nothing happens, but deny works perfectly:

Also when I check Never Ask Again, it completely disables allow button:

Not sure what is happening here, but same app works perfectly on my OnePlus2 which uses Android 6.0.1 too.

The only strange thing about my test phone is that it does not have sim card currently.

Note that both phones have Google Account attached with same synchronized contacts.

Question

I am wondering what causes this?