Taxation of UK savings interest

Article 11 of the UK-Japan Double Taxation Convention states:

Interest arising in a Contracting State and beneficially owned by a resident of the other Contracting State shall be taxable only in that other Contracting State.

So if I'm resident in Japan and earn interest from savings in a UK bank account, the UK cannot tax that interest. However, if I'm a non-permanent resident in Japan and I don't remit that money to Japan, then it seems like Japan wouldn't tax it either. So I can earn as much interest as I like and no one taxes it. Such a loophole seems somewhat improbable, so what have I missed?

Other details of this hypothetical scenario:

• I will be earning a salary from a British company paid into a British bank account. This will be Japan-source income, so I will pay income tax on this in Japan. Some of this income will be remitted into Japan.
• I will be earning rental income in the UK. This will use up my £12,570 personal allowance. This income will not be remitted into Japan.
• I will earn enough interest on my cash savings to exceed the £1000 personal savings allowance, so this interest would otherwise be taxable in the UK.