Well, Cs-137 together with Sr-90 are the major contributors to the radiotoxicity of spent fuel, at least between ~5 and ~30 years old. And it is the major source of external radiation from the waste for considerably longer time.
There are many factors, that has to be taken into account to tackle your claim. Here are a couple of examples.
One single unshielded typical LWR fuel element, will give a dose rate of 10 Sv/h at one meters distance, still after 20 years of cooling. As the lethal dose is usually taken to be 5Sv, this means, that you will probably die soon if you stand there more than half an hour. If you are cold and hug the warm element, the dose rate goes up to 100 Sv/h and you will now have only 5 minutes to get internally fried. There are hundreds of fuel elements changed in he reactor core every year.
Here is a short page on the radiotoxicity of spent fuel. So if we can trust this French governmental organisation, and look at the graph, ten year old fuel has a radiotoxicity of about 3 hundred million Sv/ton, if ingested. This means that one ton of this stuff (which is about 4 bundles) could kill more than 50 million people, if someone would, let's say, grind it up and mix it in a large batch of tomato soup cans. Calculated this way, spent fuel from one reactor core could theoretically wipe out all of humanity. You also see that the decline is pretty slow, so time helps, but not very fast. After 100 years, it has only gone down by about a factor of three.
If we calculate from that 3 hundred million Sv/ton (1e6/3E8=3.33e-3), we see that we need to ingest only 3 mg of the fuel to get a dose of one Sv, or about 15 mg to get the lethal dose of 5 Sv. This puts the waste clearly in the highest
toxicity class there is (LD50 = 5mg/kg body weight, or 350 mg for a 70 kg person).
So while the amount in tonnes is small, the amount of radioactivity or toxicity is not small at all, and high level nuclear waste is nothing to play around with. That said, I do agree that we can handle it in a safe enough way, as long as we keep our cool about it.
The first principle is that you must not fool yourself, and you are the easiest person to fool - Richard Feynman