## Thursday, October 06, 2011

### Probability

I just received Schaum's "Probability and Statistics" and other stats textbooks from Amazon. I use probability in my work but what it is? why everything works by it? Prof. Stephen Hu tries to explain probability. He says that quantum collapse (what it is?) is not necessary, probability can be explained by assuming that each time there is a decision-making event, reality splits into different universes: one universe gets decision A and the other gets decision B. To the observer in any of the universes, the effect is the same as probability. I'll consult it with Koko (her pet cat - see pic - inherited from the late Schrodiger, must know. She lives concurrently in this and infinite other universes.)

zarkov01 said...

Steve Hsu is trying to explain how probability gets used in quantum mechanics, and get around the need for that "collapse of the wave function." I think this is what is known as a "realization" in probability theory. So if X is a random variable, the event X= x is a realization of the random variable. Kolmogorov put all this on an axiomatic basis.

I haven't seen the Schaum book, but I can say that in my opinion you should get "Introduction to Probability" Vol I by William Feller. This is the best book ever written on elementary probability theory. But stay away from Vol II, which is awful. Vol I mostly deals with discrete sample spaces. The other good introductory book is the one by Papoulis which presents probability from an engineering viewpoint. A classic and very good for beginners. The statistics part should be done with a separate book. If probability is the forward problem, then statistics is the inverse problem. Hogg and Craig is a good standard introduction. If you want to go the Bayesian route than get Andrew Gelman's book.

I think a water engineer would need statistics for measurement error analysis. Hydrology uses statistics. You might also want to read up on extreme value theory. Engineers who design structures really need extreme value theory. The classic work is the one by Gumbel, but that completely obsolete. Get the book by Stuart Coles, which is very good and up to date.

Any way that's my two cents on a subject that I have experience with.

J said...

Thank you.

I use statistics in water engineering mostly to calculate optimal sizes of systems, probabilities of events such as storms, failures, and so on. Excel is enough for my needs.

J said...

Schaum contains solved problems that I use in my classes. It is not free from mistakes.

zarkov01 said...

Do not use Excel for statistical calculations. Perhaps the current version has fixed the numerous problems. But I would check before using. McCullough ran tests on many packages and Excel came out the worst. So bad he recommends against it use. http://www.forecastingprinciples.com/files/McCullough.pdf

The most accurate package, the only one that ran perfectly on all tests was Mathematica.

I recommend the R package which is available for free and is maintained by a community of users and developers. R has become the lingua franca in the academic statistics community.

I use Mathematica, but you might find it too expensive, although they sell a home version. The Mathematica people (Wolfram Research) now take statistics very seriously and version 8 has a tremendous amount of probability and statistics.

Depending on the problem, you might get a seriously erroneous result with Excel. Stay away.

Sounds like you need extreme value and reliability methods in your work. Get the Leemis book on reliability. The best introduction.

Since you consult you can't afford to make an error. In the U.S. you would need to have errors and omissions insurance. If you got a wrong number out of Excel and that caused your client to suffer, you could get sued. I would not like to have to explain in court why I ignored the warnings about Excel.

Anonymous said...

So that means that there's some alternate universe where Dr. Mengele pointed the other way and did not spare my father's life and in that universe there's no me?

K

Anonymous said...

K, you should try to put it behind you. Your family has suffered enough, I suspect.

Anon.

Anonymous said...

This is a tough thing to put behind you - to know that your grandparents and aunts and uncles and cousins have no graves, that they are just some ashes in a Polish field mixed with the ashes of a million other Jews. That if some man had twitched his thumb in another direction there would be no you (in this universe) . Easy for you to say.

K

Anonymous said...

Not so easy, actually. We also fought the war, both wars actually, at some cost.

Anon.

Anonymous said...

My father used to tell me that in his early days in America people used to approach him (if they learned he was a Holocaust survivor) and tell him things like, "you know that we had rationing - you could only get x lbs. of meat every week." This to a man who had nearly starved to death in the Nazi camps.

K

J said...

Dr Zarkov,

I have access but dont use the programs you mention. As said, Excel is more than enough for my limited needs.

I dont live in the academy and those reading my reports and designs are unable or have no interest or time to revise my calculations. On the contrary, I have found ignorance and dislike for them - judges in particular hate probabilities and equations.

The only time I witnessed someone attacked because of incorrect calculations was when my former boss Dr. Dreizin presented his Masterplan of Israel Water Economy - תוכנית אב למשק המים - and professors from the Hebrew University said and wrote that they cannot remake his calculation, it is all wrong, etc. Dreizin gave copies of this draft to criticize it before and our Math Ph D girl said it was all wrong, that the probabilities of droughts were wrong and the volumes of water reserves required were wrong, etc. I didnt criticize his maths but commented on climate change (it didnt help me). In spite of all, Dreizin's plan was approved by the Government and financed, and thus the Israeli desalination industry was born. Dreizin deserves to be honored for this feat, which is political in its essence.

I have to conclude that probabilities in my environment serve mostly to reinforce an intuitive conclusion.

zarkov01 said...

J.

Unlike engineering, the work product from statisticians requires technical ability on the part of the consumer. One does not need expert knowledge to see if a television set operates, a bridge stands up or water flows through pipes. So yes in many ways statisticians write for other statisticians, and not judges or politicians. There are exceptions of course. Insurance companies could not exist without their actuaries. If you have better actuaries, then you can price your insurance contracts more accurately than your competitors. Similarly in finance. An investment bank with better mathematicians can price derivative instruments better than the competition.

If I understand you correctly, in your environment, you might as well not do any probabilistic calculations at all except for cosmetic purposes. The numbers need not be correct, they need only seem to be correct. Nevertheless I would not use Excel for any calculations except simple arithmetic. For example, I would not trust it to get an optimization calculation right.

As for judges and politicians, you are quite right; they are loathsome creatures. In the U.S. a judge gave an order that could not be satisfied because it was mathematically impossible. The determinant was zero. When presented with this fact, the judge said something like "That is a problem for Dr. Einstein, obey the order." In the old USSR some railway engineers told their bosses that a certain train would break the rails because it was too heavy. They were sent to the Gulag for the crime of "lack of faith in Soviet technology." The next group of engineers, seeing what happened to the first, approved the new higher weight. Of course the train broke the rails. They too were sent to the Gulag for the crime of "wrecking." Survival becomes ever more difficult in an irrational world.

J said...

Fortunately for us engineers, Israeli judges and politicians have blind faith in "experts". Statistics, you are right, serve to demonstrate that you have made work and deserve your fees. One thing no one will dispute is Dr Dreizin astonishing verbal ability, he dominated the meetings with his deep voice, his oversized bald head, his fantastic store of old Hebrew roots and store of exotic Talmudic references in Aramaic. Anyway, no one can discuss statistics in a meeting. And his Masterplan was right even if the Professors were right that his numbers were wrong or made up.

Anonymous said...

Allied POW's did not exactly get well fed, and were sometimes executed. My father was held by the Germans for 2 or 3 years and was lucky to survive, on the run as an escapee for some of the time in the mountains, in winter.

POW's apart, many of the very best men of the British Empire (the officer class, in fact) were slaughtered like animals on the front lines, in their hundreds of thousands, especially in WWI.

These wars literally bleached us out, and Britain never recovered.
There was enough suffering to go round.

Anon.

J said...

I heard that the Dutch in Batavia suffered much under the Japanese occupation.

Anonymous said...

I had a schoolmaster who was ex-Royal Navy, and had served out east.

He was a mild-mannered gentleman, but point-blank refused to greet the Japanese Ambassador when he visited our school as a guest.

I also took French lessons with an elderly Dutch woman who had been captured in Singapore or Malaysia by the Japanese with her husband, a doctor, and her 2 year old son.
They murdered her husband, but she and her son managed to survive. We talked mostly about this, and as a result my French is not as good as it might have been.

Anon.

J said...

K,

The phenomenon you describe is only too human. It happened with us too. We arrived in Argentina and were wery nicely received by my uncles but one imbecile wife had to monopolize the dinner table to tell in detail her own suffering during WWII in Buenos Aires: how they feared to talk by phone, how they had to put close the shop after Evita's death, and there was a scarcity of European imports. My parents - my mother was liberated from Auschwitz only ten years before and then ten years of Stalinist dictatorship (we were class enemies) - were much affected and depressed by her marginalization of Hungary's and their own sufferings. She died after much suffering of cancer, it would be nice to be able to believe that it was God's justice.