Dalkey, near Dublin, could pass for an Irish Malibu. Both are enclaves for the most successful people in the entertainment industry; both, in their own way, elite patches of pristine natural beauty. But Dalkey isn't sunny, it's overcast and grey; the coastline betrays a melancholy texture and the locals respect the privacy of their superstar neighbours, such as the Corrs and Bono of U2. On this wet and windswept October afternoon, outside the home of Ireland's most esteemed film maker, a local points out what really matters to the Irish: "He's a novelist too, you know."

Neil Jordan is wandering around his home, which is two Georgian houses combined, in rumpled, baggy, green army fatigues and trainers, deep in thought and humming a private tune. It is a casual, warm environment; muddied shoes in various sizes are cast off under tables, and off the hall are Jordan's offices, where he writes every day. Jordan lives here with his second wife, Brenda, and their two sons, Daniel, 13, and Dashiel, 11. He has five children - two older daughters from his first marriage and a 16-year-old son, Ben, from another relationship - and all of them frequently visit or have moved back into the house in Dalkey.

Seconds later, Jordan appears in the sitting room, bashfully asks if I would like a cup of tea, and disappears again. Gargantuan windows look out over the bay and it's easy to understand why he prefers this view. Hollywood and its tentacles seems a universe away.

Neil Jordan, it is said, is not big on small talk and doesn't make direct eye contact. The impression was of a brooding and serious man, thoughtful and literary with no patience for frivolity. It takes a few minutes for him to settle in, but after some last-minute puttering, he closes the door on the bustling family, the ringing phones and lingering chores. His manner is slouchy and boyish, not intimidating at all, and it conveys depth. He looks like the man in the room you most want to talk to but would have a hard time approaching. He lights up a cigarette, looks right at me. So much for the rumour of avoiding eye contact.

Jordan is primarily known as a film maker who is unafraid to take on controversial subjects. His films often have characters that are driven by things that even they don't understand. He's perhaps best known for The Crying Game, for which he won the Oscar for best screenplay. The perception of him in Hollywood is that he's attracted to dark material - but Hollywood loves the word "dark" and will attach it to anything troublesome. Anything, he says, that's interesting is considered dark. Now, having made 14 feature films, including Mona Lisa, Interview with the Vampire, Michael Collins, The End of the Affair and The Good Thief, does he feel there is one that best represents who he is?

"This one," he says, referring to his latest film, Breakfast on Pluto. From most people, this would seem like a shrewd answer - plugging the project, it makes sense. But Jordan doesn't seem to care or be conscious of this. And as he explains the reason for this choice, it makes sense. He's never done anything "optimistic" before.

The emotional tenor of the film is new, even though it still has many of Jordan's signature imprints. There's the actor Stephen Rea, who has appeared in nine of his films. And there's also a political theme - even though it's not central to the story; it takes place in 1970s London, during a tumultuous period of conflict with the IRA. The lead character is a transvestite who, in spite of being dealt continually harsh circumstances, manages to maintain a positive outlook. There is a comic lightness to the tragedy and, even though there are similarities between this film and The Crying Game (both feature transvestites and terrorism), Breakfast on Pluto, Jordan's second adaptation from a Patrick McCabe novel (the first was The Butcher Boy), is more of a fairy tale.

In 1976, Jordan won the Guardian Fiction Prize for his first published book, a collection of stories called Night in Tunisia. Since then he has published four novels. Shade, which came out last year, was the first one in over a decade. Every day he writes, and now he has promised his publishers something new - but what?

"I've been distracted with movies," he says, brushing a tuft of hair out of his eyes. "Writing a film is very quick for me. It's a different thing because when you write a script you know that it's not finished. And you know that people will see it. People go to the cinema and they won't move until they come out. Whereas with a book, people read it, close it, put it down, come back to it six months later - or they lose it."

He rarely goes stretches when he's not writing something, but the transition between novelist and movie maker doesn't strike him as that jarring. As he sees it, fiction-writing is torture, film-making is a pleasure. Although both are authoritative, writing a novel is insular and isolated with long solitary stretches of time, lingering over words and decisions; descending inward. Directing a movie, on the other hand, requires an extroverted persona, someone able to lead and command and make a multitude of decisions with only seconds' notice. So how does he manage to reconcile the two?

"It's totally schizophrenic," he says. "When I was writing Shade, I had to stop everything. Didn't even take phone calls or travel in a plane to have a meeting. That went on for about a year. Towards the end I missed it terribly."

The fact that it was only one year reflects his level of energy. Many novelists will seal off from the world for years at a time. Novelists observe the world through novelist goggles, and Jordan seems able to wear these goggles while making movies. Asked what he feels defines him more, his response is immediate: film maker.

"I suppose it's because I've made 14 movies and written only five books," he says.

So the identification he feels is based on quantity? "Yeah, I guess so. I mean, you are what you spend your hours doing, aren't you?" That's a pragmatic way of looking at it and when I disagree, he reconsiders. "Okay, well in that case I'm a Buddhist monk."

He seems hard to pin down. It's not that he's being cagey or withholding, but there's little about himself he's willing to reveal. He's direct - he considers himself a novelist who makes films because he is from a country with a huge literary tradition. Or, as he puts it, "It's the water that people swim in here," making fiction-writing sound like a biological imperative born out of location. Yes, his work clearly comes out of the Irish storytelling tradition but still, there is an element missing. Where is his story in his stories?

Born February 25, 1950, in Sligo, Ireland, Neil Jordan grew up the second oldest of five children. His mother is a painter, his father was a schoolteacher, and he was brought up speaking Gaelic until the age of six. It was a repressive upbringing and writing was an escape. He never questioned his talent to do it; it was something he felt he could do and he felt he had no other options. "The only other option would have been to be a schoolteacher like my father, which I thought was dreadful," he says.

He got his university degree in history and English and, because he was qualified, tried teaching but he definitively states how much he despised it. "I couldn't control a class. I just hated everything to do with it, really."

He quit and started writing full time. "When I began to write there was a tremendous amount of struggle. I had tremendous battles with my father. The idea of me doing fiction frightened him. Writing about sexual matters, myself, the world I grew up in."

His life at home was not turbulent but Ireland at that time operated under religious, social and political bigotry. There was a lot of fear. "Books were censored then. Henry Miller was banned."

At 20, Jordan moved to London, where he found work as a builder and wrote; he was tearing down walls, literally and figuratively. "The first novel I ever wrote was not published. I was working in London and when the publisher sent it back, it was sent to my home in Ireland. My father read it - which he had no permission to do. And he wrote me a 20-page letter telling me how disgusting it was and I was furious."

His father died in1986 when Jordan was in Los Angeles screening The Company of Wolves. "I remember getting the call," he says. "I was meeting Princess Anne - she was at a screening of the movie. I don't know why. She was very attractive, actually. Isn't that bizarre? In that horsey, British kind of way . . . " He pauses. "So I had to go back."

When Jordan speaks of this, he is soft-spoken, matter-of-fact, with no bid for sentimentality. Which raises the question, were he and his father close? There is a long pause and he appears taken aback. Not upset with the question, just surprised. When he answers, it is at a higher volume - and with a protective authority.

"We had a relationship that was full of tremendous argument. It was a small house and I used to sit down there scribbling, and sometimes he'd come in and read it and I suppose he was supportive of the idea, but it never felt like it because there was such argument in the house.

"My father was a very intelligent man who grew up in a society which had very few options. He was very widely read. He read an enormous amount. He had a very inquiring mind, but the Catholic Church ran the educational system so someone as bright as that would only let themselves inquire so far.

"There was probably a lot of frustration in his life. Maybe that was the source of the arguments between us. Because I didn't feel the same inhibitions about expressing myself. He borrowed money off me at one stage - he wanted to buy a caravan. So he was big enough to ask me for it. And I very happily gave it to him."

When Jordan was 26 years of age, Night in Tunisia was published. The book received acclaim and launched his literary career. It was at this point that he says he felt like a real writer because he could make a living at it. There is a story in Night in Tunisia about a father who is a saxophone player in a dance band and a son who is a pianist, and the entire argument between father and son is conducted through music. It is an extension of his own relationship, and music, throughout Jordan's life, has been a constant.

"I had a relationship with my father that was deeply embedded with music. But not that kind. He was tremendously interested in classical music, and we used to go to symphony concerts together when I was young and he gave me a very good musical education. But I do remember - very, very well - the moment when I didn't want to go any more. I was about 14. And he would leave the tickets, they would be left on the mantelpiece, and one day I just didn't take one."

Jordan was always an obsessive film-goer, and he applied to the National Film and Television School but the fees were too high. Film-making seemed like a dream to him, this fantasy thing that was so far off. So he began to write scripts and one of them, Traveller, was made into an independent film. "I found a tremendous freedom in writing something that was not literature. I really found that I got excited."

He was hired as a script consultant on John Boorman's film Excalibur, released in 1981. He had written the script for Angel, which was sent to Channel 4. Ken Loach and Mike Leigh wanted to make it, but Jordan said he wanted to direct it himself. He was able to trust his instinct, to let go of the novelist and become a director, because both force you to articulate, and film was the ultimate escape.

The Miracle, which starred his then girlfriend Beverly D'Angelo, is the only film he's made based on his own work, an early short story. But even though that was a deeply personal film, he won't allow the connection to be considered autobiographical. "There's nothing personal whatsoever in it," he says. Finding where Neil Jordan is in a Neil Jordan film is a challenge. He doesn't set out to tell his own personal story but uses elements that allow him to connect to the material in an emotionally attached way.

Michael Collins was his most controversial film, particularly in the UK, amid accusations at the height of the IRA atrocities that it rewrote the history of Ireland's fight for independence and legitimised terrorism. Back then, Jordan's need to be combative and in-your-face was at its peak. The Crying Game had made the audience decidedly uncomfortable, but making Michael Collins became an exercise in constant defensive manoeuvring. And it wore him down. "Every day there were articles in the newspaper and the police came to the house. There were threats. It was about political violence - it was right in the middle of a public argument.

"For example, there was a scene with a car bombing that was taken as a direct reference to car bombings in the north of Ireland, so I had to write to the Disabled Police Officers Association saying I didn't mean any offence - it's really hard work constantly trying to explain yourself."

He doesn't get depressed very often, but admits he did while making The Butcher Boy. "The more I said, 'This is not for you,' the more they said that they, the studios, wanted to do it. They thought it was wild and wacky, but they funded the movie. I brought it to them when it was done and showed it them at the Warner Brothers lot, and I remember walking out and they were just sitting there: 'What the f*** is this thing?' They didn't even preview it. It's a very lonely feeling having made a movie and have people wonder why they made it."

And what about personally? Jordan's first marriage - which produced two daughters, Sarah, 30, and Anna, 27 - ended after little more than 10 years. Surely that must have triggered some emotional resonance? "Yeah, that was depressing," he replies. "I got depressed when I had to go into the court and sit in front of the judge and I was on my own. Yeah. It was awful."

He was married young, at 24. He fell in love with someone and wanted to live with them and at the time, you had to be married.

"I didn't particularly want to get married. But the parents would have been outraged and the thought of grandparents having heart attacks . . . "

They lived apart for many years and he moved into a house nearby so they shared the upbringing of their daughters - but there was no divorce in those days, and he had to wait until many years later. Just then, it seems as if a connection has been made - prompted by what, it's hard to tell. He has been processing a thought, and returns to an unanswered question. "The films I make are about desire," he says, turning his gaze to outside the window and staring out at the sea. "People desiring absolutes where they're not finding them, in a way."

In particular, Mona Lisa seems to address the part of his life he has just been discussing. "When I go into a studio and watch a film I made 20 years ago, it's so expressive of the person I was at that age, and I can't explain it. But everything about it is - it's about loss. I was recently separated - with two kids - and I put it into the movie when you don't even know you're doing it. The subtext of the movie was all about me."

That film, which earned Bob Hoskins an Oscar nomination, is a crime story set in London, but Jordan makes the association - vaguely - with a confused male, the Hoskins character, who drives a prostitute to her rendezvous and finds himself falling for her. But how? He reaches for and lights up another cigarette. The Hoskins character "placed emotions in totally inappropriate places and misunderstood women. That was me at the time".

It's beginning to emerge that Jordan denies that his movies are autobiographical because he thinks it's egoism. Instead, he chooses to refer to them as "highly personal". Writers, though, are always working things out on the page from their life, so there must be a progression - a personal progression - that he's noticed, especially since his latest film is more serene. There's acceptance and goodness in it.

"I'd like to make movies about happiness but it's a very boring subject. Not a lot of dramatic tension there. In a non-ironic way, I mean."

A few hours have passed and life on the other side of the door is beckoning. He looks impatient. "Yeah," he says agreeing with this assessment. "You're asking about myself and I don't spend an enormous amount of time with self-reflection." He leans over and stubs out the butt. "I'd like to give up cigarettes," he smiles.

Just when I think that's about as much self-reflection as Jordan is willing to offer, he continues. "Maybe I don't like it. Who I am. I have conversations about what I should be doing with my life. But I think of my life as my life is what I do. So I have conversations about how I can do what I do better.

"I have anxiety - that I'm not good enough. Or that I try to adhere to an image of what a film director or a writer does rather than what I myself should do. Maybe I just think I'm not good. Sometimes. It's natural, yeah? You have to have doubt. And reality can be a let-down. Sometimes. Part of me wants to do the Borgia film because I'd like to learn Italian. If I don't make the film, I'll never learn Italian."

He's referring to his next project, a film that he's written about the Borgia family. It was put on hold because he couldn't get the funding, but now it's back on track. The story, of course, is what interested him.

"Lucretia Borgia's father became Pope - he bought his way into the papacy. And it's narrated by Machiavelli. It's a great story. Drama, misuse of religion, political power. If I'd made it four years ago and it came out now, they'd all be saying it's about the Bush administration. But it's about somebody who uses religion and morality as a cloak for all sorts of things."

There are reasons why no Hollywood studio wanted to make this movie. One reason is that a lot of people don't want to offend the Catholic Church - but Jordan has never shied away from controversy and he's from a Catholic background, so he doesn't mind offending it.

Other than the film about the Borgias, he says that he would love to make a film about contemporary warfare - the way it is now. With our time coming to an end, the common thread in his films is still oblique.

"A lot of them star Stephen Rea?" he offers, laughing. Then, squirming a little, he becomes thoughtful. "I could say that a lot are about realistic beginnings and unrealistic illusions." And the same could be said about his life.