inverse_ofで解決できる
いや (追記)rejet_ifの可能性の方が大きいかも
Rails - accepts_nested_attributes_for を使った入れ子のformが罠だらけな気がする - Qiita

参考URLとその重要部分を引用する
（最も重要である回答のソースはSOを見てもらえばよいと思うので引用しない）

ruby on rails - Trouble with accepts_nested_attributes_for on validating foreign key - Stack Overflow

can't be blank while using accepts_nested_attributes_for (Rails 4) - Stack Overflow

Try removing contact_id to test if it will accept or not. If it does accept, it means you're not passing the contact_id through your params to the model

accepts_nested_attributes_for with Has-Many-Through Relations

It seemed Rails was not setting the notice attribute on the EntityRole before attempting to save it, triggering the validation errors. This is a bit surprising as other has_many relations (omitted in this blog post) should have the same save mechanics and were working just fine.

When you use collection= assignment with a has-many-through (as accepts_nested_attributes_for does), you have to specify inverse_of for Rails to save everything correctly.