Which sounds better:
A. "Insert Company Name" Press
B. "Insert Company Name" Publishing
Depends what "Insert Company Name" will be replaced by. For example, using your name alone, I prefer "Brian Publishing" over "Brian Press" but "Foster Press" over "Foster Publishing". But if I was starting a publishing business using my last name, I like "Leiper Publishing" more than "Leiper Press".
As for the rest of what you've asked, I know very little about starting a business in the UK, and none at all about it stateside.