A reticent man who has spent much of his life in the shadow of his famous younger sister, he rarely seeks the limelight. But with concern mounting about Angelina's behaviour, he has used his first-ever newspaper interview to defend her reputation and shed some light on the damaged and dysfunctional upbringing which has shaped them both.
In a fascinating insight into a very private world, he blames the bullying and manipulation of their handsome movie-hero father, Jon Voight, star of Midnight Cowboy, for blighting both their lives. And he says it is the trauma of their childhood which has driven his sister to create a huge and independent family of her own.
Already emotionally vulnerable, the death of their mother in January of this year has left 31-year-old Angelina shattered. She is eating little and already seems to be losing her famous curves. Now, more than ever, says her brother, she is determined to place her faith in the lives of the children - and who knows how many more there will be - she is gathering around her.
"I have no memory at all of my mother shouting at me or at my sister," James explains. "But I do have horrible memories of my father and the way he behaved. He was so tough on our mother.
"He lived in the same town. We saw him around Christmas time or at school recitals. He was always around but he never did his job as a father. I think one of the reasons Angelina and I care so much for other people is his treatment of our mother. Angelina and I were very protective of her for that reason."
It was their mother, Marcheline Bertrand, who provided the emotional underpinning to their lives, he says - the one point of stability amid the chaos. Her loss to cancer, aged just 56, has been devastating.
"Angie has become very thin because she's grieving," he admits. "It's even difficult for her to eat. I keep saying to her, 'Don't forget to eat'.
"But you know she doesn't pay much attention to food anyway and she's been going through a process of grief like me. She has not wanted to eat, nor has she been able to."
For all Marcheline's efforts to create a happy home, it was impossible to escape the destructive influence of Voight.
Money was one of many battlegrounds, although there should hardly have been a shortage of cash. By 1975, he was a household name, handsome, blue-eyed and in demand.
Yet according to James, it was a constant battle - to the point that Angelina and her older brother were reduced to pleading with him to pay the child support he owed. "I tried hard when I was alone with him," recalls James. "The alimony was always an issue. He would claim he was taking good care of her.
"But in reality Mum was stressing out and we had to tell him he was causing the problem. This would be when I was 13 and Angie was 11.
"We definitely had a taste of his manipulative behaviour. But it was when he dealt with my mother that we really saw him pulling the strings. And that made us very sad and angry. Angelina and I saw my mum oppressed. Now we're protective of women on their own and, of course, of children." It was an experience which drove brother and sister closer and even today their relationship borders on intensity. H for 'Haven' is tattooed on Jolie's wrist.
James, who has the same striking looks as his sister, travels with Brad and Angelina on many of their foreign expeditions, and spends hours in the company of her growing brood. Growing up, they never went hungry, but there were hardships. No one in Hollywood walks - Los Angeles is not built for the convenience of pedestrians - yet neither Angelina nor James could afford a car.
He explains: "One of the saddest things in my life happened when I was 16, which is when you can get your permit to drive in LA. I did not have a car in High School and neither did Angelina. "Try to imagine. You go to Beverly Hills High, one of the wealthiest High Schools in the nation. Even the cheapest car that anyone has is brand new. All my friends are well off. I have a movie-star father and no car.
"It was debilitating. I did not to go to the prom [the social highlight of an American high school career] because I felt uncomfortable that Dad would have to drive me. It's an embarrassing thing.
"The significant years of dating and getting to know yourself, I didn't experience any of that. I didn't experience what I think is one of the most important times in a person's life, and neither did Angelina. It affected both of us.
"When I was in college and finally dating I went to my dad and told him what it was like for me in High School. He didn't try to understand at all. He said he didn't have money."
It was a lesson that neither of them forgot. James says: "Angie has been driven to be an independently wealthy woman now because we saw what it was like to be at the mercy of someone who controls the money and pulls the strings.
"There were times when our dad was awesomely good with us, but my biggest and most abiding memory is that he would look more to our faults than to our strengths."
However, it did nothing to blunt her ambitions or her determination. In fact, the young Angelina was precocious in many ways.
This was a girl sufficiently confident to start a sexual relationship at the age of 14 and bring her lover back to live under the same roof with the tolerance, if not the approval, of her mother.
James denies that his sister is a domineering character, claiming that her famed decisiveness is, like her love of knives and her boasts about sadomasochistic sex, an attempt to be independent of other people's control. But he is not ashamed to admit he is under her influence, in all sorts of ways.
"She's always been very protective of me with regards to my romantic life, which I love," he smiles. "She wants me to be with the right woman. When you come from a divorced home, divorce is what you desperately want to avoid.
"Maybe I haven't found the right woman because my sister is too picky for me. Any woman has to go through two filters, me and then my sister. I'm a perfectionist by nature. Then, because I'm so close to Angie it's like I've already got the perfect woman in my life and it's hard for anyone else to live up to that. I go to Angie for advice and won't do anything without her because we were so close as children."
Her influence on him is so profound, he claims he has even been persuaded to adopt a child himself - provided he can find a partner first. Angelina has even brought about a change of career, persuading him to spend large parts of his time campaigning for charity. He is currently preparing to take part in a major US campaign to fight the AIDS pandemic in Uganda.
James is outraged by the claim that their relationship might in any way be inappropriate - claims prompted by 'that' kiss after the 2000 Oscars.
"I did not give Angie a French kiss, it was something simple and lovely," he insists.
"She was about to go off to Mexico to finish filming Original Sin with Antonio Banderas. I congratulated her on the Oscar win and gave her a quick kiss on the lips. It was snapped and became a big thing round the world."
Now, with the death of their mother, the wound is red raw.
"At some point I've got to be able to let my bitterness and anger towards him go and I'm working on that now. I believe my mother's in a better place. What I have to deal with is that there's no such thing as him making amends now - he's lost his chance and that just makes me and Angelina even more upset.
"When I first lost my mother, I couldn't stop crying. Fortunately, I was with Angie when we heard. It was such a big moment.
"If Angelina understood something about missing parents before, she understands completely now. Angelina and I are orphans too, in a way. When you see those kids in orphanages, they cling to each other. And that's exactly what they have - they have each other."