I think there are two reasons. One is that although farmers are few they care deeply about farm subsidies because they gain so much per person which means that they will almost all vote on the basis of that issue. By contrast, the majority of non-farmers lose so little per person that almost no one will vote on that issue, and of course many non-farmer voters don't even know about the issue.
A second reason is that a lot of people feel sympathy for farmers. Food is after all the most important product in the economy since we would all die without it, so many feel that the producers of it should be supported. This of course doesn't follow since while it is of course necessary that food is produced, any potential shortage of farmers due to too low incomes for farmers would be self-corrected on the free market with higher prices. However, that is probably how some people think and some perhaps also have a emotional/sentimental sympathy for farmers.
As a result, far from all non-farmers who care about the farm subsidy issue are opposed to it.