In 1519, the Venetian ambassador described the Cardinal's residence, York Place, as 'a very fine palace, where one traverses eight rooms before reaching his audience chamber'.

Cardinal Wolsey and Henry VIII

In 1514 Thomas Wolsey (c1475-1530) was made Archbishop of York and began work to further extend York Place, which soon became a favourite visiting place for his king, Henry VIII (1509-47).
Wolsey's unparalleled position of trust with the King began to break down in the late 1520s but before he fell from favour Henry VIII stripped him of all his assets in the south of England, including the Archbishop's town house, York Place.
Of the Cardinal's possessions York Place was the most important to the King, because the old royal palace at Westminster had been largely destroyed by fire in 1512 and Henry had been staying at Lambeth Palace as a substitute.
The acquisition of York Place, renamed Whitehall, meant that the monarchy once more had an appropriate residence in Westminster. 

Henry’s pleasure buildings

Henry VIII continued and extended Wolsey's building programme. He acquired a large plot of land opposite his new palace on the west (or park) side of the public road from Charing Cross to Westminster.
There he built for himself a series of pleasure buildings including tennis courts, a tiltyard for tournaments and a cockpit.

The main buildings of the palace, including the great hall, chapel and royal apartments, stood on the east side of the road (the side of the present Banqueting House) stretching to the river.
The street was spanned by two splendid gateways, the King Street Gate and Holbein Gate, which enabled members of the court to pass from St James's Park to the palace without crossing the public road.
On Henry VIII's death the palace covered 23 acres and was the largest royal palace in Europe. 
 

The first banqueting houses at Whitehall

The ceremonial use of Whitehall Palace demanded a number of large communal spaces for entertainment that included the great hall and the chapel, but from time to time temporary structures were constructed for special occasions. The largest of these was built by Queen Elizabeth I (1553-1603) who erected a large banqueting house to hold entertainments connected with her marriage negotiations with the Duke of Alençon in 1581. This building occupied the site of the present Banqueting House.
The Banqueting House of 1581 was probably meant to be only a temporary structure but, in fact, it continued in use for 25 years. Although the building had substantial foundations its main structure was of timber and canvas and so it must have become very dilapidated by 1606 when James I of England and VI of Scotland (1602-25) decided to replace it with a permanent building. Built of brick and stone and completed in 1609, the new banqueting house had a large hall above a ground floor basement, like its predecessor.