Because of his asthma, the Duke never held a full-time post , instead devoting his energies to building up his extensive library, which filled ten rooms and comprised over fifty thousand volumes.
After his death in 1843, the Duke's second wife, the Duchess of Inverness, continued to live at Kensington Palace until 1873.