The bank will pay US$100mln to the state and US$400mln towards relief for communities and homeowners.
RBS chief executive Ross McEwan said: "Settling these issues is a stark reminder of the heavy price we continue to pay for the global ambitions pursued by the bank in the run up to the crisis," he said.
The lender still faces a potentially hefty fine from the US Department of Justice in relation to the mortgage mis-selling saga.
RBS was expected to receive the penalty last year but no settlement was agreed, helping the bank return to a full year profit after nine years of losses.
In the full year results published last month, RBS said it had put aside £3.2bn to cover US mortgage litigation suits from the DoJ and other investigations.
Sky News reported last Friday that a deal could be agreed within weeks.
The New York attorney general's office said the sale of risky mortgage-backed securities had contributed to a crash in house price during the financial crisis.
"While the financial crisis may be behind us, New Yorkers are still feeling the effects of the housing crash," said state attorney-general Eric Schneiderman.
"Today's settlement is another important step in our comprehensive effort to help New Yorkers rebuild their lives and communities."
RBS remains more than 70% owned by the government following its bailout during the height of the financial crisis.