Using Perl Script for memory calculation of NDB Cluster in Fedora 9

Step 1: Check that you have perl

get the perl DBI module installed by using,

yum install perl-DBD-MySQL

step 2: cd into $MYSQL_HOME(or mysql base directory usually /usr/local/mysql and try

./bin/ndb_size.pl --database=myDb --user=myUser --socket=/tmp/mysql.sock --password=password

If this fails with the error like Can't use string ("9/16") as a HASH ref while "strict refs" ..... then you need to get fix the bug by replacing the following peice of code,

current code, in release ..it starts from line number 920


foreach my $i(@show_indexes)
{
$indexes{${%$i}{Key_name}}= {
type=>${%$i}{Index_type},
unique=>!${%$i}{Non_unique},
comment=>${%$i}{Comment},
} if !defined($indexes{${%$i}{Key_name}});
$indexes{${%$i}{Key_name}}{columns}[${%$i}{Seq_in_index}-1]=
${%$i}{Column_name};
}


need to be replaced as follows,

foreach my $i(@show_indexes)
{
$indexes{$i->{Key_name}}= {
type=>$i->{Index_type},
unique=>$i->{Non_unique},
comment=>$i->{Comment},
} if !defined($indexes{$i->{Key_name}});

$indexes{$i->{Key_name}}{columns}[$i->{Seq_in_index}-1]=
$i->{Column_name};
}


then executing,

./bin/ndb_size.pl --database=myDb --user=myUser --socket=/tmp/mysql.sock --password=password

should get the output for you.

Looking to optimize "Group by" in MySQL?

Few months back I was asked for the reason behind the recommendation by the consultants from MySQL. When the system was performing poorly the consultants had walked in and simply requested them to add the "order by NULL" clause to the queries. To the joy of the client and amazement of the tecnical team at the client location, this simple fix had done the trick and suddenly there was a marked improvement in performance! This had left the technical team confused, as they started to wonder why and how this happened. WhenI heard this I was confused ( as always ;) ), as I could not think of any possible logical explanation for this behaviour.

Last week while I was reading the planet MySQL I bumbed into an article talking about order by NULL and I think I had found the possible reason. Any how it is not a statement, which would optimize all queries , instead it would optimize all the statements with a group by clause.

To make things simple lets take small example,

EXPLAIN SELECT CountryCode, COUNT(*) FROM City GROUP BY CountryCode \G

This gives the following output,

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: City
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 4079
Extra: Using temporary; Using filesort

The extra column indicates that both filesort and temporary is being used, which is an indication of poor performance.

Alternatively when its applied with a Order by NULL as shown below,

EXPLAIN SELECT CountryCode, COUNT(*) FROM City GROUP BY CountryCode order by NULL \G

gives an ouput,

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: City
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 4079
Extra: Using temporary

Which indicates a bit of an improvement as it does not any longer have Using filesort.

This addition of a order by NULL could give a performance increament by many folds at times.

The next most important question is how does this happen?

MySQL by default when having a group by statement would also order by the same column, which would require an additional sorting to take place. Any how when the order by NULL is included, it does not do a sort and thereby give a performance improvement.

Sir, but with love

From my childhood I had been bought up in an atmosphere where the "Sir" was used to refer the teacher's at my school and quiet rightly these people were those for whom we had great admiration for and looked up to as exemplary characters who molded us into what we are today. It’s true that this way of referring to my masters was something that was taught to me or even one could argue that this was a protocol which was forced on to me while I was kid. Whatever the view point the reader may have I still think that while being a kid this should be the way in which the system should continue because as kids its not possible for us to distinguish an ordinary person from a personality to be referred as "Sir" and also else as a kid we would never learn to respect people. Looking from where I presently stand I still call these great teacher’s of mine as “Sir” and with the same or even more amount of admiration and respect for the part they had played in my life. Probably this mentality still is in my mind as I had already done a benchmark of this term “Sir” at a very high level and being a servant in the software industry where the term “Sir” is frowned upon.

The sudden emergence of the practice in the local universities by the “temporary tutors”, who are just a batch or two senior to demand from the rest to refer them as “Sir” or “Madam” seems ironical. It’s an acceptable practice, if it’s a military establishment where you find a culture where people salute the ranks and not the person and as long as I know a university is not a military organization as long as it’s not a military university. I find that these people had been caught up in a “psychological illusion” where they consider them to be lecturers or even in the caliber of professors now it self. The real reason for this being the fact that unfortunately for many and fortunately for some we have a serious scarcity for qualified lecturers and you find that people fresh out from the degree very often doing lectures to degree students. Please note that I am not blaming these fresh graduates for teaching as I myself had been thought by some good teachers who were fresh graduates, but I am actually throwing caution at the people who had got blown out by this new temporary seat they had been assigned to.

The only thing that I can say to them is wait for the turn and at that point people will call you as “Sir” out of respect for the wisdom you posses or the personality you are or for charisma you posses and not to jump the gun to demand or rather in a way beg for respect, remember when you get the respect that you had demanded specially from a university student its just a temporary thing, it does not come from the heart.

No pun indented, no corners spared. Its just some musings from “sayy” so take if something is worth out of what is written or even don’t be hesitative to spit on me(I mean literarily ;) ) by the way of a harsh comment if you really feel that I deserve it for writing this.

Dealing with inactive listeners – my experiences

For presenters dealing with inactive listeners is one of the biggest challenges and process of brining back inactive listeners to the active state is an art in its own right. For some campus teachers this is a night mare and at the end of the day you find them depressed and really disturbed after a teaching session, when they feel that they had not succeeded in getting there point across to the audience. Some of the seasoned profs have chosen either of the two extremes, one extreme group, believes that, whatever they say they expect the students to listen or at least pretend to be listening by nodding, having eye contact and all, the other extreme group doesn’t really care about the audience at all and just preach whatever they want and doesn’t really care if they had got their point across or not.

The objective of this piece of writing is to make the presenter to be more objective in their expectations and analysis of there audience and thereby becoming more successful in getting there message across to their audience.

The Four major reasons for people seeming to be or really being inactive listeners

1. Environment
2. Cultural differences
3. Incompetence of presenter
4. Inherent human weaknesses

Environment

Awareness of the environment in which you are doing the presentation is of utmost importance; you should know who is participating in the session and have a general idea about what would be their knowledge in the topic that you are talking about.

Further more consider the physical factors like the following, is the stage properly organized? Is the projected content visible to everyone in the audience? Is the air-conditioning temperature at appropriate level? If using a microphone check and see if it’s correct? etc

When talking about the environment setup my mind goes back to my school days, when I was a senior student at college I used to go to my primary school to teach the kids to play chess, incidentally I was teaching in the same class that I had studied about ten years before. I was teaching the kids and suddenly I noticed a student, who was sitting in the same seat I used to sit years before, being the worst listener. I walked behind his seat casually and looked out of the window to know what was distracting him and then I remembered that my teacher, when I was the kid in that seat, always made it a point to have the window curtains down before she started the class and never told me the reason why it should be closed. At that time I can remember that this was something I hated to do, because the class was overlooking a lush green grass yard which was the playing ground for the monkeys and I wanted to watch them and closing the curtain was preventing me from doing what I liked, but only when I stood in the shoes of a teacher I realized her wisdom and from that day onwards I too made it a point to have the curtains down in that classroom before I started the sessions.

Culture

Understanding the cultural behavioral attitude of people is very important to understand the listeners. Just because a person does not look into the presenter’s eye that does not mean that the particular person is not listening to you.

A classic example of this happened on my first international class I did, I had audiences from an array of communities and there were a few ladies from cultures where they are not used to look in to the eyes of a man. I noticed that they never looked up at me and were focusing on the computer in front of them or at the book. Initially I was worried as I was clueless as I was not sure if they had understood what I was talking about. I played a bit of a mind game with them to get them out of there usual behavior. I first got them to look at the content that was being projected on the screen, by showing irresistible content which is pertinent to the audience. This was a confidence building exercise, although it was a slow process I managed to bring them to a state where they lifted the face to look at the projector and at this time I showed a lot of energy in my presentation and in a few minutes time, when they had enough confidence in my behavior they were behaving like any other audience. The lesson learnt was that when it comes to measuring listeners, we should not use the common measurement for all people; we should appreciate the uniqueness of each individual.

Incompetence of Speaker

This has two major categories, one where you find the speaker is incompetent due to his lack of knowledge in the topic he is speaking about or lack of preparation, the second major category is where the audience has a premature feeling that the speaker is incompetent due to his outer look or age.

In order to convince that the presenter has enough competence the presenter will have to make sufficient preparation on the areas he is going to talk about and show a lot of energy and commitment, which could make the audience, change their initial perception on the presenter.

I can remember on a few occasions when I was introduced as an expert in a certain domain the people looked at me in a very sarcastic way because of my age, any how I knew this was the main challenge that I will have to face and was ready for it. Like a typical test batsman I would put the head down and bat and the only way in which I could show that I could prove myself was my proving that I have the knowledge although at a lesser age than a typical expert.

Inherent Weaknesses

Be informed that any human being will loose focus when you are doing a long session and its quiet natural for people to switch off. Think from the perspective of the listener if you were the listener how long will you be able to keep focus on a single task. The most challenging thing for a presenter is to master the art of facilitating the return of an inactive listener or a listener who had lost focus, when you find that a listener is making an effort to return to the presentation, make it a bit easy by doing a quick recap on what you are doing or talking about. Especially if you had given a hand out point out the slide you are now in.

Most of the reasoning for the inactive listening resolves around the above four categories.

After identifying the problem we should consider the possible solutions;

I provide below a few handy tips to minimize inactive listening.

Solutions

Hint 1:

Drive the entire audience as a team; use each others energy to sustain the interest among the audiences.

Hint 2:

Relate to things people closely associate with, for example there job.

Hint 3:

Use the natural competitive instinct of people.

Hint 4:

Ask question that you know the users know the answer to ease them up and make them to be more at home.

Hint 5:

Use Humor, but remember this should be good clean humor and always know that not all people are 100% receptive to humor. Refrain using humor involving race, colour, religion and gender.

Hint 6:

Encourage the right type of participation, when someone asks the right type of question cohesive to the content you are talking about, explain the question to the audience and then answer the question.

Hint 7:

Spur up some irresistible topic which will make them get involved.

Hint 8:

Spring a few surprises relevant to the content.

Hint 9:

Manage the time well.

Hint 10:

Make it look as if the thing that you are talking about is the simplest thing on the planet. Specially, when you want to make a product presentation or a teaching lesson.

*** Incidentally the above was the main content one of the presentations I did to my colleagues at office at their own request and I can feel that it had made a paradigm shift in the way they did and dealt with people during the presentation.

watch out - md5 sum bug

The other day I was in for a rude surprise when it was reported that in the application which we had developed, the authentication at times fails. The application required to auto generate an md5 encrypted value for authentication and it was found that md5 value generated was dropping the leading zero's when ever the actual md5 sum value was supposed to have leading zero's.

It was a rude shock for me.

This was the piece of code that I had,

String unencryptedValue= "test"; MessageDigest m= MessageDigest.getInstance("MD5"); m.update(unencryptedValue.getBytes(),0,unencryptedValue.length());
String encryptedValue = new BigInteger(1,m.digest()).toString(16);

Then it took me some time to realize that the BigInteger casting was causing the havoc, and at the same time waruna gave me a link to the following blog http://jdwyah.blogspot.com/2006/08/java-md5-password-hash.html, which has the solution for the above problem,

its just a matter of introducing another method which would pad the leading zero's to the md5 value to make up the required length.

the fixed code looked something like below,

String unencryptedValue= "test"; MessageDigest m= MessageDigest.getInstance("MD5"); m.update(unencryptedValue.getBytes(),0,unencryptedValue.length());
String encryptedValue = new BigInteger(1,m.digest()).toString(16);
encryptedValue = fixLeadingZeroBug(encryptedValue, 32, '0');

public String fixLeadingZeroBug(String s, int length, char pad) { StringBuffer buffer = new StringBuffer(s); while (buffer.length() <> buffer.insert(0, pad); } return buffer.toString(); }

It was just now a matter of fixing the bug and instead I learnt an important lesson to be a mindful of these leading zero's when ever an integer is used in place of having it as a String for intermediate processing. But unfortunately I had to learn it the hard way. :(

Factory Method

The third in the series was the factory method, was rather a simple pattern to present. Any how for the last two weeks (thanks to vanky) I have been having the Head first pattern book with me. This provided me with a nice peace of example which I could use for the presentation.

I had a big surprise at the presentation when I suddenly found that the presentation that i was making was being recorded on video. Probably I should be more careful when i make my next one as I leaving evidence of all the lies I say at the presentation.... haha

Composite pattern

I had the opportunity of doing my second presentation of the Design pattern series of presentations which my self and Vidula were doing alternatively. This time it was the Composite pattern that I had to present on. Got a bit too busy and the slides which I created for the purpose was something I didn't feel proud about, it was a bit incomplete and I just scrapped through explaining the essentials of the pattern in a half an hours time.

And then came the question of the day! Harsha presented the audience and myself with a question to try.

He simply asked us to use the composite pattern to solve a search problem????

"The question"

Suppose you have a gui like follows,
How is it possible to use the composite pattern to create the where clause of the sql statement dynamically?

It was a nice question to make us think about the composite pattern and its practical application.

This in turn led to an interesting discussion on topic.

The solution to this, as explained by romith, was to have the AND and the OR as the nodes and the criteria like the Name , City etc as the leaves!

Initially I thought that the Name, City and NIC could be made into nodes and have the AND and OR as the leaves.It was quiet rightly pointed out by Harsha that it would work , but was not the correct answer, as it was not a scalable solution and having it the other way around would provide with an extensible and a scalable solution.

I ended this presentation with a lot of self satisfaction as it was a presentation that I left with a clear picture about the topic than the idea that I had at the start of it! But ironically I was the presenter.. haha poor audiance!