Light:

Get yourself some sort of work light, the better lit your work area is the easier it will be to work. I picked up a small florescent light fixture for under $30 bucks, and it works great.

Digital Multi-Meter:

Any multi meter will do that can measure DC voltage, AC voltage, resistance and current. I use a $200 meter at work and it does the same job that the meter that i got at the local hardware shop on special for $15. The only difference is that the one we use at work is weatherproof, do high 600V testing (which you will not need) and can take a 3-story fall.

Flush Cutters:

You can pick up a pair for a few bucks. They are great for working with electronics.

Wire strippers:

There are many styles of Wire strippers. Stay away from the automatic strippers (especially the butterfly style). I use to build electrical panels for a living, so I use to strip a lot of wire, and Klein Wire Cutters are THE BEST, its worth spending the extra bit of money for conform. As a hobbyist you will work with smaller wire gages (the higher the AWG# the smaller the wire). I find 22AWG a really nice gage to work with.

Breadboard:

A breadboard is a prototype board, it is used to prototype circuits without having to solder components together, A must-have.

Soldering Iron:

You can go without a soldering as long as you do not want to build anything permanent. When you do decide to spend some money buy the AOYUE 2900. Its 70 watt, it heats up fast (2-5 seconds), the tips last a long time, the tips are inexpensive (and contain the heater element, so when you replace the tip you also replace the heater). The problem with cheap irons is that they are not meant to last, the heater and the tips are only good for 20-40 hours, and then your stuck replacing them, and for the price of 5 cheap irons you could own this one which holds its own against irons that cost 5 times as much.

Tweezers:

At some point, your going to have to align a small part (or hold one in place) its $5 that will save you a lot of frustration.

Jewelers Screw Driver Set:

You could pick up a set for $5-$20. I like the larger screw plastic body sets over the thin metal sets, but its a personal preference, so get a set that feels comfortable to use.

Needle Nose Pliers:

Pick up a set that feels nice in your hand. I have 2 sets on my desk, one set has a spring that lets me operated it with one hand, and the other does not. I suggest picking up one of each.

Small Clamps:

You could pick up some small clamps for really cheap at the dollar store. Get twice as much as you think you will need. No matter how many you have you will find yourself needing more.

Tooth Brushes:

Get some rough tooth brushes from the dollar store, they are great for cleaning up your circuit boards after soldering.

Other:

As you move on you will add more tools to your toolbox but you don't need much more than what is included here to get started. I, for one, love my Oscilloscope, but I don't recommend anyone who is starting out buying one.

Sparkfun has a great Beginners Tool Kit if you don't have any tools this is a great set. If you eventually decide to build your own boards from scratch you'll need a AVR ISP programmer, but if you only plan on sticking though the Arduino you won't need it

Solder:

There many types of solder on the market, The easiest to use is Leaded 63Sn/37Pb with a water-soluble flux core, Although the no-clean stuff sounds good don't use it, it continues to slowly eat at the board over time and requires worse chemicals than any other flux type to fully clean off. Leaded 60Sn/40Pb with a Flux core is the most common solder found in hardware stores. A good high quality solder will make soldering a lot easier. I do not recommend Lead-Free (Silver) solder, it has a higher melting point and does not flow as smoothly as Lead solder, in short, its a pain in the ass to use on a good day.

Flux:

Flux is basically a mild acid that you use to clean your soldering surfaces before soldering. Using flux makes soldering much easier, the downside is that you need to clear the flux off once you are done soldering or the acid will continue to eat at your board. I highly recommend a Water soluble Flux, Standard flux requires Isopropyl Alcohol to clean.

Wick:

We all make mistakes, and sometimes being able to de-solder something is just as important to solder something. Out of all the devices made for removing solder the wick is by far the easiest and best (aside from an expansive hot air rework station).

Wire:

Wire is rated in AWG (American Wire Gage) the higher the number the thinker the wire (basically this number states how many times a wire goes through a roller in order to get its thickness, so a 30 AWG wire went through a roller 30 times). Use 26-30 AWG is a nice thickness for electronics applications. Wire also comes in 2 varieties Stranded and Solid core. Solid core is good for breadboards and Stranded is better for soldering. A wide array of colors is recommended. I recommend Red (for positive power), Green(or white for ground) and Black (for Signals). There are many different types of Insulation and any will do for electronics.

Isopropyl Alcohol:

Used to clear Flux of your board after soldering.

Electric Tape:

It always comes in handy, the shiny stuff is the cheap stuff, avoid it.

Heat Shrink:

Heat shrink is great, It's basically a tube that shrinks to half it sizes when heated, this is great for shielding wire after you have spliced 2 wires together. I use a lighter to shrink heat-shrink (hold the heat shrink ABOVE the flame and turn slowly). Get the 1/8 inch diameter stuff, that's good for most of the hobbyist applications.

Jewelers Screw Driver Set:

You could pick up a set for $5-$20. I like the larger screw plastic body sets over the thin metal sets, but its a personal preference, so get a set that feels comfortable to use.

Needle Nose Pliers:

Pick up a set that feels nice in your hand. I have 2 sets on my desk, one set has a spring that lets me operated it with one hand, and the other does not. I suggest picking up one of each.

Small Clamps:

You could pick up some small clamps for really cheap at the dollar store. Get twice as much as you think you will need. No matter how many you have you will find yourself needing more.

Tooth Brushes:

Get some rough tooth brushes from the dollar store, they are great for cleaning up your circuit boards after soldering.

Other:

As you move on you will add more tools to your toolbox but you don't need much more than what is included here to get started. I, for one, love my Oscilloscope, but I don't recommend anyone who is starting out buying one.

Sparkfun has a great Beginners Tool Kit if you don't have any tools this is a great set. If you eventually decide to build your own boards from scratch you'll need a AVR ISP programmer, but if you only plan on sticking though the Arduino you won't need it

WinAVR:

This is the C compiler and associated software for the AVR. All of my tutorials have been written using this software.

AVR Studio 5 or AVR Studio 4

This is the IDE developed by ATMEL.

AVR DUDE:

The AVR Downloader / Uploader. This is that we are going to use in order to upload the code to your AVR over USB once a Bootloader installed on the AVR.

xLoader:

The Software that lets you send .hex files to the AVR via the Arduino Bootloader.

Arduino:

The Arduino is a wicked piece of kit, it's a simple board with an AVR (at this time the ATmega328), all the voltage hookups and a bootloader that basically allows it to be programmed using a USB port, no extra programmer required! WOOT for saving money!

Resistors:

Resistors are the most common and versatile component used in electronics. 1/8 watt resistors are inexpensive and will work for most applications, but if you could get 1/4 watt resistors for the same cost nab them up, any rating higher than 1/4 Watt is used for special applications thus are more expansive. 270,333,1k,4.7k and 10k are the most common used resistor values.

Capacitors:

Every project that you will work on will have a capacitor in it somewhere so make sure you pick up a variety pack.

Push Buttons:

The easiest device to hook up to an input. "Tac Switches" are fairly cheap and come in all shapes, colors and sizes, use what you like.

LEDs:

An LED is the simplest device that you could connect to an output, and let's be honest, we all love flashing lights. A mix of various colors of Diffused type 5mm LEDs (red, green and yellow).

Voltage Regulator:

A great little device, that well... regulates the input voltage into a given value. In simpler words, most devices work on 3.3V or 5V the voltage regulator reduces the power of a standard 9V battery to your desired voltage.

Transistor:

While you're not going to need them right away, they do come in handy. Transistors have many uses in electronics. trust me, add a few 'small signal' PNP and NPN transistors to your electronics order, you'll be glad you did because eventually you will need them.

Diode:

Diodes are another really common component used for many things, they are really cheap so make sure you keep a few around. The 1N4001 diode is a nice generic choice.

Over time your parts shelf will grow larger and larger, but the above list is a great place to start.