Chances are you will have heard of Hay-on-Wye. You might even have been. It’s the town on the Anglo-Welsh border where more than 30 years ago a man called Peter Florence began what has become the world’s most famous literary festival.

Now some 200,000 people descend on the place each May and June, and for 11 days it feels like the centre of the literary universe, with hordes carrying tote bags traipsing hither and thither and pubs and restaurants overflowing like Venice in high summer.

If that’s what floats your boat, then get stuck in. But for my money, Hay is worthy of a visit in its own right – and preferably when all those other visitors (not to mention ex-presidents and Booker prize-winners) aren’t there. When all the fuss of the festival has subsided, you’ll be able to appreciate this doughty little market town for what it is. (You’ll also stand a far better chance of getting a table at the Old Black Lion, regarded as one of its best local restaurants – though you’ll still need to book.)

Famously, Hay has some 30 bookshops, mainly second hand and antiquarian, often specialising in slim portion of the market. This means that Hay is to books what Florence is to Renaissance art, and on a hot August day the trickling Wye can easily exceed the stolid green Arno in beauty.

The Uffizi among these emporia is Booth Books. Timbered, tall – more like Liberty in London than any bookshop you’ve ever seen – it was founded in 1961 by Richard Booth, who would later proclaim himself the King of Hay. Where he led, others followed, and by the 1970s Hay had earned its first literary reputation – as a town of bookshops.

These days there are several other generalist booksellers to compete with Booth’s, as well as specialists such as Murder and Mayhem – a crime bookshop where you can find the complete works of Agatha Christie in five or more paperback editions – and the chic-looking Poetry Bookshop. Among the generalists is Green Ink Booksellers – tagline: ‘intellectual recycling’ – which also claims to be the town’s newest second-hand bookshop.

But the bookshops are just part of Hay’s charm. Rising above the town is the castle. In parts Norman, Tudor and Jacobean, it is now in the care of a local trust, having previously been owned by Richard Booth, of whom there is a memorial bronze in the grounds.

The castle has survived fires and dereliction and has now emerged, sandblasted and pristine-looking, from plastic and scaffolding after ten years of renovation. As of this summer it is gloriously open to the public for the first time in its 900-year history – with viewing platform, café, shop with esoteric books and an art gallery. (The castle also claims to have the oldest wooden door in Britain, dating back to the 1340s. Don’t miss it.)

Sitting with coffee on the terrace overlooking the town is a joy – beyond the roofs are trees and green hills galore. Tuscany is never this lush. Down below is the pillared butter market – a kind of Georgian version of a Roman temple to saturated fat – which was rebuilt in 1833 by one William Enoch.

A decent stone’s throw away is another castle, a 10th century motte-and-bailey grassy hillock about the size of a Saxon burial mound, built by the first Normans to arrive. It’s next to the car park of the livestock market and is easy to miss, but don’t. It’s worth the stinging nettles and brambles to climb to the top of this mound and feel like Bernard de Newmarch, the first Norman lord of Hay.

But the real delight is the town itself. Get lost among the timber-framed buildings and Georgian stone and all sorts of things pop out at you. There’s the solicitor’s, Gabbs, the sort of name Dickens would have chosen for a profession where they charge and punish by the word. Then there’s the Victorian Gothic clocktower, chiming the hour melodiously as it has since the 1880s; and Goldsworthy’s, the sort of mid-market outdoors shop-cum-gents outfitter that is a Jurassic survivor in the modern world of e-commerce. Here you can buy an air rifle, a Thermos flask, ladies’ handbags, shoes and expensive woollen jumpers in August. I bought suede shoes and stocked up on 1.77 calibre pellets.

Across the road is Eric L. Pugh & Co – in understated Art Deco lettering – which has an arrangement of Roberts radios in the window like Edward VIII is partying all over again. They might sell iPhones or mobile covers inside, but I hope not.

In the other direction is The Cosy Café, on the corner down the hill, where brunch comes on sourdough and the furniture is mid-century modern like it’s Dalston or Brooklyn. If that doesn’t float your avocado, in the other direction there’s Shepherds, a bustling local institution with curved glass front windows that are a wormhole through time. Step inside for a pot of tea and teacakes, a coffee or an ice cream.

Even the local Londis is remarkable; through its large Victorian windows I saw shelves heaving with fresh vegetables and loaves of freshly baked bread that wouldn’t have looked out of place in France. A few shops down there’s Bartrums, a stationer, in an elite shade of dark brown-green. Here you can find five or six types of scissors and notebooks galore, or acquire what might be the world’s most desirable pencil sharpener in bronze.

Around the corner, the faded jade and white paint and net curtains thicker than Beijing’s internet firewall announce what might be saddest looking Chinese takeaway this side of Suez. But try their chop suey – hot Chinese food to take away rarely tasted better. Then for breakfast in a bun on the run you have Angie’s, where I can confirm you’ll find the lady herself cheerfully behind the counter.

The best place to eat in in town is the tapas bar, the whitewashed Tomatitos, with its red bunting outside. Go in and order one of everything from the blackboard and watch the tables turn. This little spot has pace and the burble of voices in here is uniquely Hay.

And before your day is over, you should trundle down past the Three Tuns, serving beer and more since the 1600s, and cross the bridge over the Wye for the sunset that I insist can go 15 rounds with the Arno on the right day. On the far side there’s Offa’s Dyke, or a footpath that follows it a least. Named after the 8th century Mercian overlord who probably built it, this is the Welsh Marches’ version of Hadrian’s Wall and once separated the Saxon realm from the kingdom of Powys.

Look back at Hay from here and see the roofs and castle high on the hill, and suddenly this bastion of books makes sense. Do go to the Hay Festival if you wish – but then go back to see this town for itself, and make Hay while the sun shines.

Alec Marsh is editor-at-large at Spear's magazine and is the author of Rule Britannia and Enemy of the Raj.