I think you are too hard on yourself.
You took some risks at very difficult tasks. What you are trying to do is incredibly difficult. Making money off agriculture requires lifelong training, and a big skill set.
You took a risk and left what you were good at, and tried a new thing. It turns out that you are not succeeding at this. It happens all the time to people when they take risks or change careers. You can’t know until you try.
I’m especially amazed the criticism is that your Japanese isn’t good enough. It’s very difficult for adults to learn a new language!
You want to go back to what you’re good at–but your wife doesn’t want to. You don’t say why it is.
You might want to try and think of a compromise. Write out the options (1) get a job you don’t like (2) move back to your previous country, etc. Then try and figure out a compromise.
You tried some difficult things, and took some chances. You did not give up immediately. You are finding out there are obstacles and limits. So you want to change your plans. This is 100% normal. Maybe your wife will understand, maybe she won’t. But you should not be hard on yourself about it. Learn from it, and move on and think about what to do differently in the future.