Thursday, October 18, 2012

My CloudWatch Monitor for AWS

Amazon EC2 offers the CloudWatch service to monitor cloud instances as well as load balancers. While this service comes at some cost (0,015$/hour/instance) it offers useful infrastructure metrics about the performance of your EC2 infrastructure. While there are commercial and free tools out there which provide this service, you might not want to invest in them or add another tool to your monitoring infrastructure. This post will provide step-by-step guidance on how to extend your monitoring solution to retrieve cloud metrics. The code sample is based on the free and open-source dynaTrace plugin for agent-less cloud monitoring. Some parts however have been simplified or omitted in tutorial. The major parts that are missing in this sample are dynamic discovery of EC2 instances and an algorithm which is a bit more reliable and accurate in retrieving monitoring data.

Step 1 – Basic Infrastructure

So let’s get started by setting up our basic infrastructure. First we need to download the Java Library for Amazon CloudWatch . Alternatively you can create your own Web Service stubs or simply use the REST interface. For simplicity we rely on the ready-to-use library provided by Amazon. The we create a Java class for our cloud monitor which implements the basic functionality we need. For brevity I will omit any imports needed – in Eclipse CTRL – SHIFT – O will do the job :-)
public class CloudWatchMonitor {
 
 private static class MeasureSet{
   public Calendar timestamp;
   public HashMap<String, Double> measures = new HashMap<String, Double>();
 
   @Override
   public int compareTo(MeasureSet compare) {
     return (int) (timestamp.getTimeInMillis() - compare.timestamp.getTimeInMillis());
   }
 
   public void setMeasure(String measureName, double value) {
     measures.put(measureName, value);
   }
 
   public Set<String> getMeasureNames() {
     return measures.keySet();
   }
 
   public double getMeasure(String measureName) {
     return measures.get(measureName);
   }
 }
 
 private String instanceId;
 
 private AmazonCloudWatchClient cloudWatchClient;
 
   public static void main(String... args) throws Exception {
   CloudWatchMonitor monitor = new CloudWatchMonitor("<instanceName>", Credentials.accessKeyId, Credentials.secretAccessKey);
   for (;;) {
     MeasureSet measureSet = monitor.retrieveMeasureSet(measureNames);
     if (measureSet != null) {
     printMeasureSet(measureSet);
   }
   Thread.sleep(60000);
   }
 }
 
 public CloudWatchMonitor(String instanceId, String accessKeyId, String secretAccessKey) {
   cloudWatchClient = new AmazonCloudWatchClient(accessKeyId, secretAccessKey);
   this.instanceId = instanceId;
 }
}
So what have we done? We defined the CloudWatchMonitor which will contain all our logic. The main method simply queries every minute for new measures and prints them. We have chosen an interval of one minute as CloudWatch provides accuracy to one-minute intervals. Additionally, we defined the inner MeasureSet class which represent a set of measures collected for a given timestamp. We have used a HashMap to make the implementation more generic. The same is true for the retrieveMeasureSet method which takes the measures it retrieves as an input. Finally we defined the constructor of our monitor to create an instance of an AmazonCloudWatchClient – this is supplied by the Amazon library we use -  and store the instanceID of the EC2 instance to monitor.  The  accessKeyID and secretAccessKey are the credentials provided for your Amazon EC2 account.

Step 2 – Retrieve Monitoring Information

Now we have to implement the retrieveMeasureSet method which is the core of our implementation. As there are quite a number of things we have to do, I will split the implementation of this method into several parts. We start by creating a GetMetricStatisticsRequest object which contains all information which data we are going to request. First we set the namespace of the metrics which in our case is AWS/EC2 (in case we want to retrieve load balancer metrics it would be AWS/ELB). Next we define which statistical values we want to retrieve.  Then we define the period of the monitoring data. In our case this is one minute. If you want aggregated data you can specify any multiple of 60. Then we define which measure aggregates we want to retrieve. CloudWatch offers average, minimum and maximum values. As our aggregation will only contain one data point all of them will be the same. Therefore we only retrieve the average.
public MeasureSet retrieveMeasureSet(ArrayList<String> measureNames) throws AmazonCloudWatchException, ParseException {
 
  GetMetricStatisticsRequest getMetricRequest = new GetMetricStatisticsRequest();
  getMetricRequest.setNamespace("AWS/EC2");
  getMetricRequest.setPeriod(60);
  ArrayList<String> stats = new ArrayList<String>();
  stats.add("Average");
  getMetricRequest.setStatistics(stats);
  ArrayList<Dimension> dimensions = new ArrayList<Dimension>();
  dimensions.add(new Dimension("InstanceId", instanceId));
  getMetricRequest.setDimensions(dimensions);
Next we have to define the time frame for which we want to retrieve monitoring data. This code looks a bit complex simply because we have to do some number formatting here. CloudWatch expects the time in a special format and all date values in ISO 8601 format which use UTC and looks like this 2010-04-22T19:12:59Z. Therefore, we have to get the current UTC time and format the date strings in the proper format.  We take the current time as the end time and the start time is 10 minutes back in the past. Why are we doing this? The reason is that CloudWatch data is written asynchronously and the latest metrics we will get will be a couple of minutes in the past.  If we set the start time to one minute in the past we would not get any metrics.
 String dateFormatString = "%1$tY-%1$tm-%1$tdT%1tH:%1$tM:%1$tSZ";
 GregorianCalendar calendar = new GregorianCalendar(TimeZone.getTimeZone("UTC"));
 calendar.add(GregorianCalendar.SECOND, -1 * calendar.get(GregorianCalendar.SECOND));
 getMetricRequest.setEndTime(String.format(dateFormatString, calendar));
 calendar.add(GregorianCalendar.MINUTE, -10);
 getMetricRequest.setStartTime(String.format(dateFormatString, calendar));
Additionally we have to add the following code to the constructor to calculate our UTC offset and define the timezone member field.
  TimeZone zone = TimeZone.getDefault();
  timeOffset = zone.getOffset(new Date().getTime()) / (1000 * 3600);
The next thing we have to do now is retrieve the actual metrics. As we will get more than one measurement we have to store them to later on select the latest measurement. The inconvenient part here is that the CloudWatch API does not allow us to retrieve more than one timer at once.  Therefore we have to make a request for each metric we want to retrieve.  Additionally you will notice some possibly cryptic date parsing and calculation.  What we do here is parse the date string we get back from Amazon and create a calendar object.  The tricky part is that we will have to add (or subtract) the offset of our current timezone to UTC.  The formatter is defined as private DateFormat formatter = new SimpleDateFormat(“yyyy-MM-dd’T'HH:mm:SS’Z'”);
HashMap<Long, MeasureSet> measureSets = new HashMap<Long, MeasureSet>();
for (String measureName : measureNames) {
  getMetricRequest.setMeasureName(measureName);
  GetMetricStatisticsResponse metricStatistics = cloudWatchClient.getMetricStatistics(getMetricRequest);
  if (metricStatistics.isSetGetMetricStatisticsResult()) {
    List<Datapoint> datapoints = metricStatistics.getGetMetricStatisticsResult().getDatapoints();
    for (Datapoint point : datapoints) {
    Calendar cal = new GregorianCalendar();
    cal.setTime(formatter.parse(point.getTimestamp()));
    cal.add(GregorianCalendar.HOUR, timeOffset);
    MeasureSet measureSet = measureSets.get(cal.getTimeInMillis());
    if (measureSet == null) {
      measureSet = new MeasureSet();
      measureSet.timestamp = cal;
      measureSets.put(cal.getTimeInMillis(), measureSet);
   }
   measureSet.setMeasure(measureName, point.getAverage());
}
The last part is to retrieve the latest available measurements and return them. Therefore we will simply sort the measurements and return the latest one.
 ArrayList<MeasureSet> sortedMeasureSets = new ArrayList<MeasureSet>(measureSets.values());
 if (sortedMeasureSets.size() == 0) {
   return null;
 } else {
   Collections.sort(sortedMeasureSets);
   return sortedMeasureSets.get(sortedMeasureSets.size() - 1);
 }
In order to make sorting work we have to make the MeasuresSet implement comparable
private static class MeasureSet implements Comparable<MeasureSet> {
 
  @Override
  public int compareTo(MeasureSet compare) {
    return (int) (timestamp.getTimeInMillis() - compare.timestamp.getTimeInMillis());
  }
 
  // other code omitted
}

Step 3 – Printing the Results

Last we have to print the results to the console.  This code here is pretty straightforward and shown below.
public static void printMeasureSet(MeasureSet measureSet) {
  System.out.println(String.format("%1$tY-%1$tm-%1$td %1tH:%1$tM:%1$tS", measureSet.timestamp));
  for (String measureName : measureSet.getMeasureNames()) {
    System.out.println(measureName + ": " + measureSet.getMeasure(measureName));
  }
}

Step 4 – Defining the Metrics to Retrieve

Our code is now nearly complete the only thing we have to do is define which metrics we want to retrieve. We can either pass them as command-line parameters or explicitly specify them. CloudWatch supports the following parameters for EC2 instances:
  • CPUUtilization
  • NetworkIn
  • NetworkOut
  • DiskReadBytes
  • DiskWriteBytes
  • DiskReadOperations

Step 5 – Visualizing and Storing the Data

As you most likely do not want to look at the data on your console, the final step is to visualize and store the data. How to implement this depends on the monitoring infrastructure you are using- Below you can see a sample of how this data looks in dynaTrace.
CloudWatch-based Instance Monitoring
CloudWatch-based Instance Monitoring in dynaTrace

Conclusion

Building your own CloudWatch monitoring is pretty easy. The metrics provided enable an initial understanding how your EC infrastructure is behaving. These metrics are also input to the Amazon EC2 Auto Scaling infrastructure.
If you want to read more articles like this visit dynaTrace 2010 Application Performance Almanac

Friday, October 12, 2012

Best way to edit the system.out.println messages

Suppose your application is a legacy application and system.out.println() is used everywhere and without any date and formatting, now you want to display the date for each message that consoles...

How can you handle it without changing the existing code ..!!!!!???

Got the clue ?? Yes?? then no need to read the below text .. you close this site and go happily.. ;) :)

You are still with me ???? then read the small trick that helps you to format the console output with out changing the code ...


try {
    System.out.println("This should go to stdout");

    PrintStream original = System.out;
    System.setOut(new PrintStream(new OutputStream() {
                StringBuilder buffer = new StringBuilder();
                public void write(int b) {
                       if ((char)b != '\n'){
                          buffer.append((char)b); }
                       else {
                       logger.info(TODAY_DATE_TIME+b);}
                }
            }));
    System.out.println("This should go to the write method defined abouve!!!");

    System.setOut(original);
    System.out.println("this should go to stdout");
}
catch (Exception e){
    e.printStackTrace();
}
Output of the effect of the above change looks like this: 
[2012-Oct-13 12:28:19] INFO [REDIRECTED] - This should go to the write method defined abouve!!!
[2012-Oct-13 12:28:19] INFO [REDIRECTED] - Starting the Trap Receiver..
[2012-Oct-13 12:29:22] INFO [REDIRECTED] - This should go to the write method defined abouve!!!
[2012-Oct-13 12:29:22] INFO [REDIRECTED] - Starting the Trap Receiver..
12:29:46: t /krishna/RD/Arcsight/log/output.log
[2012-Oct-13 12:28:19] INFO [REDIRECTED] - This should go to the write method defined abouve!!!
[2012-Oct-13 12:28:19] INFO [REDIRECTED] - Starting the Trap Receiver..
[2012-Oct-13 12:29:22] INFO [REDIRECTED] - This should go to the write method defined abouve!!!
[2012-Oct-13 12:29:22] INFO [REDIRECTED] - Starting the Trap Receiver..


[2012-Oct-13 12:34:55] INFO [REDIRECTED] - ****** New Trap Received *******
[2012-Oct-13 12:34:55] INFO [REDIRECTED] - Received at: Sat Oct 13 12:29:22 IST 2012                                                                                             
[2012-Oct-13 12:34:55] INFO [REDIRECTED] - From: 127.0.0.1/51929                                                                                                                 
[2012-Oct-13 12:34:55] INFO [REDIRECTED] - PDU: V1TRAP[reqestID=0,timestamp=0:00:00.00,enterprise=1.3.7.9.45.89,genericTrap=0,specificTrap=0, VBS[1.3.6.1.2.1.1.4.0 = Rajakrishna Reddy]]                                                                                                                                                                         
[2012-Oct-13 12:34:55] INFO [REDIRECTED] - securityName: public                                                                                                                  
[2012-Oct-13 12:34:55] INFO [REDIRECTED] - securityModel: 1                                                                                                                      
[2012-Oct-13 12:34:55] INFO [REDIRECTED] - securityLevel: 1                                                                                                                      
[2012-Oct-13 12:34:55] INFO [REDIRECTED] -                                                                                                                                       
[2012-Oct-13 12:34:55] INFO [REDIRECTED] - Processed 45.45454545454545/s, total=1   
FUN to be coding!!

Getting Local IP of a system in java


The Best way to get the local IP of a system in Java?????

/**
* Gets the local IP.
*
* @return the local ip
* @throws UnknownHostException
*             the unknown host exception
* @throws SocketException
*             the socket exception
*/
public static String getLocalIP() throws UnknownHostException, SocketException
{
String ip = InetAddress.getLocalHost().getHostAddress();
final Enumeration<NetworkInterface> ifaces = NetworkInterface.getNetworkInterfaces();
for (NetworkInterface iface : Collections.list(ifaces))
{
final Enumeration<InetAddress> raddrs = iface.getInetAddresses();
for (InetAddress addr : Collections.list(raddrs))
{
if (!addr.isLoopbackAddress() && addr.isSiteLocalAddress())
{
ip = addr.getHostAddress();
}
}
}
return ip;
}


Do you think is there any other way!??? It should work in all the scenarios ... !!

Thursday, October 11, 2012

Shell script Tips - Using functions and Executing a java process as background one

It is small shell script which teach you about the things, that we use in our routine shell scripting life.
They are:
1) Functions.
2) Resolving the current path ro absolute path.
3) Blinking Text.
4) Proper usage of log.
5) Executing a java process in back ground.
6) Usage of ps and grep commands.
7) Proper code formation.

#!/bin/bash

########
# This is to log the events to a file in events directory, monitors and manages the java process which
# actually reads  the files and sends the events to third party for processing. 
#
# It also demonstrates the usage and handling of certain features defined in *ix kernal.
#
# @author Rajakrishna V. Reddy
# @version 1.0
#
########

### CONSTANTS may be changed over a time ###
jarName=arcsight-integration-1.0.jar
javaLogFileName=javaProcess.log
scriptLogFileName=script.log
javaFileName=com.mt.arcsight.ArcsightManager
NOW=`date +"%y-%m-%d %H:%M:%S"`


### Variables to be defined or automatically get defined by script as it is flowing ###
myName=$(readlink -fn $(type -p $0))
currentDir=$(dirname $myName)
eventDir=$currentDir/../events
logDir=$currentDir/../log
confDir=$currentDir/../conf
binDir=$currentDir
eventFileName=
event=
keyValuePairLength=

### Logging the text in blinking mode, and set it back to normal mode after display ###
function blinkingLog
{
        echo -e "[$NOW] [ArcsightServer] [INFO]: $* \033[0m"
}

### Creates all the required dirs, if doesn't exists ###
function makeAllDris
{
makeDir $eventDir
makeDir $logDir
makeDir $confDir

eventDir=$(cd $currentDir/../events && pwd)
logDir=$(cd $currentDir/../log && pwd)
confDir=$(cd $currentDir/../conf && pwd)
}

### Creates a dir, if doesn't exists ###
function makeDir
{
if [ ! -d $1 ]; then
#echo "Events dir is not exists, going to create: $eventDir";
mkdir $1
#echo "Created the event dir successfully: $eventDir";
return 1;
fi
}
### Just check and create the event dir ###
function checkAndCreateEventDir
{
if [ ! -d $eventDir ]; then
#echo "Events dir is not exists, going to create: $eventDir";
mkdir $eventDir
#echo "Created the event dir successfully: $eventDir";
return 1;
fi
}

### Just check and create the log dir ###
function checkAndCreateLogDir
{
        if [ ! -d $logDir ]; then
blinkingLog "\033[31m Log dir does not exists, \033[32mgoing to create it: $logDir"
                mkdir $logDir
                blinkingLog " Created the event dir successfully: $logDir";
        fi
if [ ! -f $logDir/$javaLogFileName ]; then
blinkingLog "\033[31m Log dir does not exists, \033[32mgoing to create it: $logDir/$javaLogFileName"
touch $logDir/$javaLogFileName;
chmod +w $logDir/$javaLogFileName;
       blinkingLog " Log file created successfully: $logDir/$javaLogFileName";
fi
}

### Get the present events count and get the desired filename by using the next event number ###
function checkAndCreateNextEventFile
{
fileCount=`ls -ltr $eventDir | wc -l`;

#Contains total number of files + "total 0", hence it gets the number of files + 1, which will be our next file number;
#fileCount=`expr $fileCount - 1`;

eventFileName=$fileCount.'event';
#echo "The next event file name is: $eventFileName";
touch $eventDir/$eventFileName;
chmod +w $eventDir/$eventFileName;
#echo "Event file is created successfully: $eventFileName."
}

### Write the content into the specified file, decided by checkAndCreateNextEventFile function. ###
function writeAnEvent
{
#blinkingLog "\033[05m Writing the given event into $eventDir/$eventFileName";
#blinkingLog "\033[05m Writing the event: $event"
checkAndCreateEventDir
checkAndCreateNextEventFile
echo "$event" > $eventDir/$eventFileName;
}

### Manages the java process by checking and starting it if not already running ###
function manageJavaProcess
{
        checkAndCreateLogDir
        pid=`ps -ef | grep -i $javaFileName | grep -v grep | awk '{print $2}'`
        if [ -z "$pid" -o "$pid" == " " ]; then
                blinkingLog " Checking the status of java app: \033[31m Process: $javaFileName is down..!!! \033[32mstarting it.!"
                java -cp $PATH:$currentDir/arcsight-integration-1.0.jar com.mt.arcsight.ArcsightManager $confDir/arcsight.properties > $logDir/$javaLogFileName 2>&1 &
sleep 0.1
        pid=`ps -ef | grep -i $javaFileName | grep -v grep | awk '{print $2}'`
blinkingLog "\033[32m Process $javaFileName is started with pid: $pid";
        else
                blinkingLog " Checking the status of java app: \033[32m Process $javaFileName is running with pid: $pid";
        fi
}

### Starting method of your process ###
function main
{
event=$*
keyValuePairLength=$#

makeAllDris
blinkingLog "\033[35m **** Sending the event to MWatch. ****";
manageJavaProcess
writeAnEvent
blinkingLog "\033[35m **** Sent the event to MWatch successfully. ****";
}

### Starting point of your process ###
main $*

Tuesday, October 9, 2012

Unix in a nut shell-1


1) Getting the Absolute Path in the shell script:

scriptFile=$(readlink -fn $(type -p $0))
echo "Absolute path: $scriptFile"

1) Reading ALL the command-line params in the shell script:

Shell script accepts parameters in following format... 
$1 : first 
$2 : second....so on upto  9th param,
$9 : 9th param 

whereas $0 : gives script/function name 

If your script has more than 9 params then !???? :( :( 

No worries ... just access them in following way... 
${12} : 12th param 
${18} : 18th param

Mastering in Linux "history" command

Tips to get mastered in using history in *ix systems!!
When you are using Linux command line frequently, using the history effectively can be a major productivity boost. In fact, once you have mastered the 15 examples that I’ve provided here, you’ll find using command line more enjoyable and fun.

1. Display timestamp using HISTTIMEFORMAT

Typically when you type history from command line, it displays the command# and the command. For auditing purpose, it may be beneficial to display the timepstamp along with the command as shown below.
# export HISTTIMEFORMAT='%F %T '
# history | more
1  2008-08-05 19:02:39 service network restart
2  2008-08-05 19:02:39 exit
3  2008-08-05 19:02:39 id
4  2008-08-05 19:02:39 cat /etc/redhat-release

2. Search the history using Control+R

I strongly believe, this may be your most frequently used feature of history. When you’ve already executed a very long command, you can simply search history using a keyword and re-execute the same command without having to type it fully. Press Control+R and type the keyword. In the following example, I searched for red, which displayed the previous command “cat /etc/redhat-release” in the history that contained the word red.
# [Press Ctrl+R from the command prompt,
which will display the reverse-i-search prompt]
(reverse-i-search)`red': cat /etc/redhat-release
[Note: Press enter when you see your command,
which will execute the command from the history]
# cat /etc/redhat-release
Fedora release 9 (Sulphur)
Sometimes you want to edit a command from history before executing it. For e.g. you can search for httpd, which will display service httpd stop from the command history, select this command and change the stop to start and re-execute it again as shown below.
# [Press Ctrl+R from the command prompt,
which will display the reverse-i-search prompt]
(reverse-i-search)`httpd': service httpd stop
[Note: Press either left arrow or right arrow key when you see your
command, which will display the command for you to edit, before executing it]
# service httpd start

3. Repeat previous command quickly using 4 different methods

Sometime you may end up repeating the previous commands for various reasons. Following are the 4 different ways to repeat the last executed command.
  1. Use the up arrow to view the previous command and press enter to execute it.
  2. Type !! and press enter from the command line
  3. Type !-1 and press enter from the command line.
  4. Press Control+P will display the previous command, press enter to execute it

4. Execute a specific command from history

In the following example, If you want to repeat the command #4, you can do !4 as shown below.
# history | more
1  service network restart
2  exit
3  id
4  cat /etc/redhat-release

# !4
cat /etc/redhat-release
Fedora release 9 (Sulphur)

5. Execute previous command that starts with a specific word

Type ! followed by the starting few letters of the command that you would like to re-execute. In the following example, typing !ps and enter, executed the previous command starting with ps, which is ‘ps aux | grep yp’.
# !ps
ps aux | grep yp
root     16947  0.0  0.1  36516  1264 ?        Sl   13:10   0:00 ypbind
root     17503  0.0  0.0   4124   740 pts/0    S+   19:19   0:00 grep yp

6. Control the total number of lines in the history using HISTSIZE

Append the following two lines to the .bash_profile and relogin to the bash shell again to see the change. In this example, only 450 command will be stored in the bash history.
# vi ~/.bash_profile
HISTSIZE=450
HISTFILESIZE=450

7. Change the history file name using HISTFILE

By default, history is stored in ~/.bash_history file. Add the following line to the .bash_profile and relogin to the bash shell, to store the history command in .commandline_warrior file instead of .bash_history file. I’m yet to figure out a practical use for this. I can see this getting used when you want to track commands executed from different terminals using different history file name.
# vi ~/.bash_profile
HISTFILE=/root/.commandline_warrior
If you have a good reason to change the name of the history file, please share it with me, as I’m interested in finding out how you are using this feature.

8. Eliminate the continuous repeated entry from history using HISTCONTROL

In the following example pwd was typed three times, when you do history, you can see all the 3 continuous occurrences of it. To eliminate duplicates, set HISTCONTROL to ignoredups as shown below.
# pwd
# pwd
# pwd
# history | tail -4
44  pwd
45  pwd
46  pwd [Note that there are three pwd commands in history, after
executing pwd 3 times as shown above]
47  history | tail -4

# export HISTCONTROL=ignoredups
# pwd
# pwd
# pwd
# history | tail -3
56  export HISTCONTROL=ignoredups
57  pwd [Note that there is only one pwd command in the history, even after
executing pwd 3 times as shown above]
58  history | tail -4

9. Erase duplicates across the whole history using HISTCONTROL

The ignoredups shown above removes duplicates only if they are consecutive commands. To eliminate duplicates across the whole history, set the HISTCONTROL to erasedups as shown below.
# export HISTCONTROL=erasedups
# pwd
# service httpd stop
# history | tail -3
38  pwd
39  service httpd stop
40  history | tail -3

# ls -ltr
# service httpd stop
# history | tail -6
35  export HISTCONTROL=erasedups
36  pwd
37  history | tail -3
38  ls -ltr
39  service httpd stop
[Note that the previous service httpd stop after pwd got erased]
40  history | tail -6

10. Force history not to remember a particular command using HISTCONTROL

When you execute a command, you can instruct history to ignore the command by setting HISTCONTROL to ignorespace AND typing a space in front of the command as shown below. I can see lot of junior sysadmins getting excited about this, as they can hide a command from the history. It is good to understand how ignorespace works. But, as a best practice, don’t hide purposefully anything from history.
# export HISTCONTROL=ignorespace
# ls -ltr
# pwd
#  service httpd stop [Note that there is a space at the beginning of service,
to ignore this command from history]
# history | tail -3
67  ls -ltr
68  pwd
69  history | tail -3

11. Clear all the previous history using option -c

Sometime you may want to clear all the previous history, but want to keep the history moving forward.
# history -c

12. Subtitute words from history commands

When you are searching through history, you may want to execute a different command but use the same parameter from the command that you’ve just searched.
In the example below, the !!:$ next to the vi command gets the argument from the previous command to the current command.
# ls anaconda-ks.cfg
anaconda-ks.cfg
# vi !!:$
vi anaconda-ks.cfg
In the example below, the !^ next to the vi command gets the first argument from the previous command (i.e cp command) to the current command (i.e vi command).
# cp anaconda-ks.cfg anaconda-ks.cfg.bak
anaconda-ks.cfg
# vi  !^
vi anaconda-ks.cfg

13. Substitute a specific argument for a specific command.

In the example below, !cp:2 searches for the previous command in history that starts with cp and takes the second argument of cp and substitutes it for the ls -l command as shown below.
# cp ~/longname.txt /really/a/very/long/path/long-filename.txt
# ls -l !cp:2
ls -l /really/a/very/long/path/long-filename.txt
In the example below, !cp:$ searches for the previous command in history that starts with cp and takes the last argument (in this case, which is also the second argument as shown above) of cp and substitutes it for the ls -l command as shown below.
# ls -l !cp:$
ls -l /really/a/very/long/path/long-filename.txt

14. Disable the usage of history using HISTSIZE

If you want to disable history all together and don’t want bash shell to remember the commands you’ve typed, set the HISTSIZE to 0 as shown below.
# export HISTSIZE=0
# history
# [Note that history did not display anything]

15. Ignore specific commands from the history using HISTIGNORE

Sometimes you may not want to clutter your history with basic commands such as pwd and ls. Use HISTIGNORE to specify all the commands that you want to ignore from the history. Please note that adding ls to the HISTIGNORE ignores only ls and not ls -l. So, you have to provide the exact command that you would like to ignore from the history.
# export HISTIGNORE="pwd:ls:ls -ltr:"
# pwd
# ls
# ls -ltr
# service httpd stop

# history | tail -3
79  export HISTIGNORE="pwd:ls:ls -ltr:"
80  service httpd stop
81  history
[Note that history did not record pwd, ls and ls -ltr]


Monday, October 8, 2012

Easiest way to take Backup and Restore MySQL Database.!


mysqldump is an effective tool to backup MySQL database. It creates a *.sql file with DROP tableCREATE table and INSERT intosql-statements of the source database. To restore the database,  execute the *.sql file on destination database.

Using mysqldump, you can backup a local database and restore it on a remote database at the same time, using a single command. In this article, let us review several practical examples on how to use mysqldump to backup and restore.

For the impatient, here is the quick snippet of how backup and restore MySQL database using mysqldump:
backup: # mysqldump -u root -p[root_password] [database_name] > dumpfilename.sql

restore:# mysql -u root -p[root_password] [database_name] < dumpfilename.sql

How To Backup MySQL database

1. Backup a single database:

This example takes a backup of sugarcrm database and dumps the output to sugarcrm.sql
# mysqldump -u root -ptmppassword sugarcrm > sugarcrm.sql

# mysqldump -u root -p[root_password] [database_name] > dumpfilename.sql
The sugarcrm.sql will contain drop table, create table and insert command for all the tables in the sugarcrm database. Following is a partial output of sugarcrm.sql, showing the dump information of accounts_contacts table:
--
-- Table structure for table `accounts_contacts`
--

DROP TABLE IF EXISTS `accounts_contacts`;
SET @saved_cs_client     = @@character_set_client;
SET character_set_client = utf8;
CREATE TABLE `accounts_contacts` (
`id` varchar(36) NOT NULL,
`contact_id` varchar(36) default NULL,
`account_id` varchar(36) default NULL,
`date_modified` datetime default NULL,
`deleted` tinyint(1) NOT NULL default '0',
PRIMARY KEY  (`id`),
KEY `idx_account_contact` (`account_id`,`contact_id`),
KEY `idx_contid_del_accid` (`contact_id`,`deleted`,`account_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
SET character_set_client = @saved_cs_client;

--
-- Dumping data for table `accounts_contacts`
--

LOCK TABLES `accounts_contacts` WRITE;
/*!40000 ALTER TABLE `accounts_contacts` DISABLE KEYS */;
INSERT INTO `accounts_contacts` VALUES ('6ff90374-26d1-5fd8-b844-4873b2e42091',
'11ba0239-c7cf-e87e-e266-4873b218a3f9','503a06a8-0650-6fdd-22ae-4873b245ae53',
'2008-07-23 05:24:30',1),
('83126e77-eeda-f335-dc1b-4873bc805541','7c525b1c-8a11-d803-94a5-4873bc4ff7d2',
'80a6add6-81ed-0266-6db5-4873bc54bfb5','2008-07-23 05:24:30',1),
('4e800b97-c09f-7896-d3d7-48751d81d5ee','f241c222-b91a-d7a9-f355-48751d6bc0f9',
'27060688-1f44-9f10-bdc4-48751db40009','2008-07-23 05:24:30',1),
('c94917ea-3664-8430-e003-487be0817f41','c564b7f3-2923-30b5-4861-487be0f70cb3',
'c71eff65-b76b-cbb0-d31a-487be06e4e0b','2008-07-23 05:24:30',1),
('7dab11e1-64d3-ea6a-c62c-487ce17e4e41','79d6f6e5-50e5-9b2b-034b-487ce1dae5af',
'7b886f23-571b-595b-19dd-487ce1eee867','2008-07-23 05:24:30',1);
/*!40000 ALTER TABLE `accounts_contacts` ENABLE KEYS */;
UNLOCK TABLES;

2. Backup multiple databases:

If you want to backup multiple databases, first identify the databases that you want to backup using the show databases as shown below:
# mysql -u root -ptmppassword

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| bugs               |
| mysql              |
| sugarcr            |
+--------------------+
4 rows in set (0.00 sec)
For example, if you want to take backup of both sugarcrm and bugs database, execute the mysqldump as shown below:
# mysqldump -u root -ptmppassword --databases bugs sugarcrm > bugs_sugarcrm.sql
Verify the bugs_sugarcrm.sql dumpfile contains both the database backup.
# grep -i "Current database:" /tmp/bugs_sugarcrm.sql
-- Current Database: `mysql`
-- Current Database: `sugarcrm`

3. Backup all the databases:

The following example takes a backup of  all the database of the MySQL instance.
# mysqldump -u root -ptmppassword --all-databases > /tmp/all-database.sql

4. Backup a specific table:

In this example, we backup only the accounts_contacts table from sugarcrm database.
# mysqldump -u root -ptmppassword sugarcrm accounts_contacts \
      > /tmp/sugarcrm_accounts_contacts.sql

4. Different mysqldump group options:

  • –opt is a group option, which is same as –add-drop-table, –add-locks, –create-options, –quick, –extended-insert, –lock-tables, –set-charset, and –disable-keys. opt is enabled by default, disable with –skip-opt.
  • –compact is a group option, which gives less verbose output (useful for debugging). Disables structure comments and header/footer constructs. Enables options –skip-add-drop-table –no-set-names –skip-disable-keys –skip-add-locks

How To Restore MySQL database

1. Restore a database

In this example, to restore the sugarcrm database, execute mysql with < as shown below. When you are restoring the dumpfilename.sql on a remote database, make sure to create the sugarcrm database before you can perform the restore.
# mysql -u root -ptmppassword

mysql> create database sugarcrm;
Query OK, 1 row affected (0.02 sec)

# mysql -u root -ptmppassword sugarcrm < /tmp/sugarcrm.sql

# mysql -u root -p[root_password] [database_name] < dumpfilename.sql

2. Backup a local database and restore to remote server using single command:

This is a sleek option, if you want to keep a read-only database on the remote-server, which is a copy of the master database on local-server. The example below will backup the sugarcrm database on the local-server and restore it as sugarcrm1 database on the remote-server. Please note that you should first create the sugarcrm1 database on the remote-server before executing the following command.
[local-server]# mysqldump -u root -ptmppassword sugarcrm | mysql \
                 -u root -ptmppassword --host=remote-server -C sugarcrm1
[Note: There are two -- (hyphen) in front of host]