The government made a promise to preserve free TV licences for over-75s, but “that promise lies in tatters”, the shadow culture secretary has said.
Tom Watson told MPs the policy “staves off poverty, isolation and loneliness”, but was now in jeopardy.
Responsibility for funding licences was passed from the government to the BBC in 2015 and the corporation has been consulting on its future.
The culture minister said the decision was part of a “fair deal” for the BBC.
It was taken when the country was in “a very difficult fiscal situation” and all government departments had to save money, Margot James added.
A licence currently costs £154.50 a year for colour TV.
The benefit was originally introduced in November 2000, when the Department for Work and Pensions agreed to pay the government directly for TV licences for those over the age of 75.
George Osborne made the decision as chancellor,to hand responsibility for it to the broadcaster.
The government’s contribution to the scheme is being phased out by 2020 and the BBC has been consulting on what to do after that.
Last year a BBC-commissioned report said pensioners were now less likely than any other age group to live in poverty, and last month, peers said “outdated” age-specific benefits, like free licences, should be replaced with support for the young to “deliver a fairer society”.
Opening the Labour-led debate on Wednesday, shadow culture secretary Tom Watson told MPs that in 2017 the Conservatives had promised to keep all pensioner-age benefits during the duration of the current Parliament.
That, he said, included concessionary TV licences for over-75s.
Mr Watson told MPs the BBC had been put in an “impossible position” by the government, where it would either have to choose between cutting programmes, or to end a benefit for older people.
“This is austerity by the back door,” he told those in the Commons, citing Age UK analysis which said ending the benefit could push 50,000 pensioners into relative poverty.
He said TV viewership among those socially isolated was a “lifeline” to the outside world, and four-and-a-half million older people “could be betrayed unless the government acts”.
A number of MPs questioned why the government was “outsourcing” a social security concession to the BBC.
But former culture secretary John Whittingdale said free TV licences were “not a fundamental pillar of the welfare state”.
He said that when the benefit had been introduced in 2000, it cost the government £365m, and as of last year, the cost to the BBC was £660m – a significant chunk of the corporation’s total budget.
Responding from the government benches, Ms James said the BBC was “a treasured institution” and a “constant companion” for many older people.
She said that when the BBC agreed to fund these TV licences, the BBC promised to consult on any further changes to them.
The government has unfrozen the cost of the TV licence for the first time since 2010, she said, meaning that the BBC was now able to charge for their broadcasting in line with inflation.
The results of the consultation launched by the BBC had not yet been announced, she added, saying that the government was taking the issue of loneliness “seriously”.
Parliament voted in favour of handing responsibility of the benefit over to the BBC, she reminded MPs.
Who has to pay?
Some residents in care homes are eligible for a reduced TV licence fee of £7.50, and those registered as blind can have a reduction of 50% in the cost.
Businesses in the hospitality sector can also get reduced rates on TV licences for their premises. People with a black and white TV only pay £52.
In 2016, the “iPlayer loophole” was removed, meaning that even those watching BBC programming on catch-up still had to pay the fee.
The BBC collects about £3.5bn annually from the licence fee.