I have a client. Her husband was a second-generation American, a Yale-educated lawyer who worked in the family business that was started by his father, a Russian immigrant. Four years ago he was diagnosed with cancer. He put up a great fight, but cancer won, and a year later he was gone at age 66.

He left $100 million, which went to his wife, son and daughter (the kids are in their late 20s). I recently had a meeting with the family. The son’s wife was a few days away from giving birth to a baby girl. As the son and I were talking about his upcoming fatherhood, I asked him what kind of father he wants to be.

“I don’t want to be like my father,” he said.

I was a bit surprised and asked why.

“After my father passed away, his friends would tell me how he was this larger-than-life, gregarious man. I never saw that man. My father worked 16 hour days, seven days a week. He worked in the basement – he’d come up for dinner and go back down. He never spent time with me or my sister. My mom did everything, from driving us to school to taking me to football practice. I always felt like I was raised by my mother. I don’t want to be like that. I want to be there for my kids.

“Until the last moment my father thought that he’d beat the cancer. So he never expressed his true feelings to me or my sister. A year later my father’s friend told me that my father confided in him that he wished he’d spent more time with us.”

Listening to him, I felt a sudden urge to run home and hug my children. I also felt enormous sadness. What if he had worked eight or maybe even ten hours a day instead of 16 and had left his kids a $10-million pile rather than $100 million? Would it really have made a difference for his kids’ lives? They are wonderful, thoughtful young adults who don’t have pretentious lifestyles. His son would probably trade all his money for a father who was there for him.

It’s not how much money you are going to leave your children; it’s the memories you’ll leave them with.

As a father who runs a business, I was deeply impacted by this story. I asked myself, am I doing the same thing to my kids?

I shared my concerns with a good friend. He has a struggling website design business that hasn’t moved out of the startup phase, ten years on. He’s anything but wealthy, but his bills are paid and his family doesn’t go hungry. He works eight hours a day and spends all of his free time with his three young kids.

As I was mentally comparing these two dads, my definition of what success means got completely redefined. It’s not how much money you are going to leave your children; it’s the memories you’ll leave them with.

His kids may not have the fancy toys and big houses that some of their friends do. But to them those things won’t matter much; our appreciation of material things has a very short shelf life. Instead they’ll have the warm glow of love their father gave them which is worth exponentially more.

Sometimes we view work like a real-life version of Candy Crush, where money is not a currency that buys us material stuff but chips that we never intend to spend but are just there to keep count of our successes. They are the currency that moves us to the next level, and the next level. Just as we can mindlessly spend tens of hours playing Candy Crush, our work can turn from being something we do in order to live into an addiction.

I was incredibly fortunate to discover my love for investing when I was in my early twenties, but I realize that there is a danger in this love which at times can conflict with my love for my family.

My own father often quoted from The Little Prince, by Antoine de Saint-Exupéry: “You become responsible, forever, for what you have tamed.”

We always want more. But this “more” has a cost that we don’t always see: time with our family. It’s my core responsibility to provide for my family, but at some point I need to say that more is not worth it.