Arduino, an Introduction to the IDE...

Unless otherwise stated, all references to the Arduino IDE relate to version 1.8.0 or later. Always keep your IDE up to date to ensure that the latest features, libraries and board definitions are being used!

The Arduino range of micro-controller boards are an excellent way to enter the world of programming and electronic control/sensing. Many Arduino owners use them for things such as robotics or home automation but there are so many uses for an Arduino that I don't think anyone has been able to list them all yet.

Arduino boards mainly use micro-controller devices made by ATmel and designated "AVR" which stands for "Alf (Egil Bogen) and Vegard (Wollan)'s RISC processor". Alf Egil Bogen and Vegard Wollan were working at Nordic VLSI in Norway when they conceived the design of the AVR devices and the company grew into Nordic Semiconductor. The architecture was sold to Atmel (another Norwegian company) and the AVR, as we know it today, was born. There are non-AVR boards in the Arduino range but we won't be discussing these here.

The Arduino range is described here.

Arduino has its critics, especially about the Editor included in the Arduino IDE but, I think many critics miss the point. The Arduino IDE is provided to enable a relatively easy introduction to the world of Arduino, programming and the hardware. This aim is satisfied by the Arduino IDE as it allows code writing, compiling and programming of the AVR devices with very little additional interaction.

Where the Arduino IDE really falls short is documentation! Like many other Open Source projects, Arduino totally fail to provide proper documentation. Most of the information as to how the IDE works and the contents of the multitude of support files is out there but it has to be sought. There is no single repository of technical information about the IDE and its use or adaption for bespoke projects.

Believe me, this is not uncommon as too many authors write wonderful programs, applications and libraries and then totally forget to write user guides. This can leave users bemused and frustrated so, come on Arduino, have a thought for your users and publish proper documentation for the Arduino IDE and hardware!

The Arduino IDE authors began changing the way the IDE works in a significant way when IDE 1.5.0 was released. IDE 1.0.x had been the last major "C" change but, at least, things did remain fairly stable during its life. IDE 1.5.0 introduced a major new approach with the introduction of Platforms and Processor specification but, on the upside, allowed users more control over the compiling, uploading and bootloading processes.

OK, IDE 1.5.x was really a "beta" version but, in my opinion, it should never have been released as, with the coming of IDE 1.6.x everything changed completely again. Unfortunately the authors decided to change their minds part way through the development of IDE 1.6.x by changing the internal command format within the "platform.txt" file.

What this means is that the "platform.txt" file for IDE 1.8.0 is incompatible with IDE 1.5.0. through to IDE 1.6.11. Any user, like me, who made necessary changes to their "platform.txt" files on the older IDE 1.5.0 suddenly found that the modified "platform.txt" files did not work from IDE 1.6.11 onwards.

What the authors should have done is update the IDE version to 1.7.0 when the mid 1.6.x changes were introduced in my opinion but, they did not.

Everything on this site to do with the IDE will relate to IDE 1.8.0 onwards unless, of course, Arduino change the internal formats again!

The First Steps...

Before getting down to working with Arduino boards we need a way to write programs and publish the code to the AVR device. To do this we need a number of tools, the most important of which are these:

  1. An editing tool to write the source programs
  2. A compiler/linker tool to convert the source code into a form which the AVR device on the Arduino board can understand
  3. A programmer to write the finished program to the AVR device on the Arduino board

The most common way of writing programs for an Arduino board, especially for the beginner,  is to use the Arduino IDE (Integrated Development Environment) within which we would normally use the C/C++ language. Remember that this high level language is for our benefit to make our programs readable to humans because the AVR device on the Arduino board cannot understand C/C++. It is the job of the compiler to convert C/C++ into the machine code that the AVR can understand.

The Arduino IDE is based on the Wiring Development Platform (IDE) and concept by Hernando Barragán. Here are the main windows of the two IDEs which clearly show the similarities (click on thumbnail to enlarge image):

Wiring IDE Window
The Wiring IDE Main Window
Arduino IDE Window
The Arduino IDE Main Window

Although the Wiring IDE is designed for use with Wiring boards, the IDE can be used with Arduino boards as well but, the Arduino IDE is more advanced nowadays and should be used in preference. Full details of Wiring can be found here. The Arduino IDE is not the only IDE available but it is the best IDE to get started with and many users never use anything else although using an external editor such as Notepad++ is a sensible move. Here are the Arduino IDE's Pros and Cons:

Pros:

Cons:

The Arduino IDE is available for a variety of platforms, Windows, Linux and Mac OS X but I will only be dealing with the Windows version here. The IDE can be installed into the Windows environment or as a stand-alone or portable application.

The Compiler...

The compiler takes the sketch that you so carefully wrote in C/C++ and performs a number of functions but basically it converts the code you write in the editor into binary (machine code) that the AVR device understands.

The above is a simplification but it roughly outlines what the compiler does. As stated before, the MCU does not understand C/C++ functions so they have to be translated into machine code which is binary and will comprise Bytes or Words (8bit or 16bit or even longer) commands which the MCU's processor understands as commands and data.

Each type of MCU or Microprocessor can have (and probably will) have completely different internal command languages and it is therefore necessary to have a different compiler for each. The Arduino IDE uses a compiler named "avr-gcc" which means that is for ATmel AVRs where C/C++ has been used as the high level programming language.

AVRDUDE...

AVRDUDE (AVRDownloadUploaDE)is a very special application which is specifically for ATmel AVRs. It performs the function of transferring the HEX file created by the compiler to the AVR device. AVRDUDE is told where the HEX file is located, what AVR device is to be programmed and how the AVR fuses are to be set/reset (during bootloading). It then takes over and communicates with the AVR device using the AVR device's protocol.

One of the first actions AVRDUDE performs is to ask the AVR what type it is by requesting the AVR to report its internal type ID which normally consists of three or four Bytes. For instance, the ATmel ATmega328P AVR has the ID 0x1E950F and this is checked against the AVRDUDE device definition file. If the AVR ID does not match, the programming function is aborted. If everything is OK, AVRDUDE proceeds to upload the HEX file to the Programmer and then read back the uploaded binary data to check that the process worked.

When a board such as the Arduino UNO has a sketch uploaded to it AVRDUDE is also used to communicate with the ATmega328P on the board. The built-in USB UART and the bootloader code on the AVR act as the programmer.

The Programmer...

A Programmer is the hardware/software that actually puts the binary result of the compiling onto the AVR device. The ATmega328P AVR uses the SPI (Serial Peripheral Interface) to transfer data into its Flash memory and, as we normally communicate using the USB or Serial interface of our PC, the Programmer provides the necessary hardware interface to go from USB to SPI. Uploading to the AVR via a built-in Serial UART is just the same except that the bootloader file on the AVR replaces the SPI interface. We send the binary data via AVRDUDE which in turn communicates with the built-in UART, The UART transfers the data over the AVR's Serial interface and the bootloader places the data into Flash memory.

Installing a "Portable" IDE instead of the Windows Installer...

The simplest, fuss free, way to install the Arduino IDE onto your Windows computer is to use the Windows Installer which is an executable file. This method will integrate the IDE with Windows and automatically set the location of the Arduino environment and files. This approach is fine if you never want to do anything other than program Arduino boards using the built-in libraries or those added using the Libraries Manager. However, this basic approach can cause problems if the IDE is updated with a new release. The IDE's Preferences file is placed in the user's workspace in "C:\Users\<username>\AppData\local\arduino15\" and a Projects folder is created in the user's "My Documents" folder as here "C:\Users\<username>\Documents\Arduino".

All working data files are saved to the user's personal folders, "C:\Users\<username>\AppData\Local\arduino15". Added "Packages" are also stored in the user's personal folders along with the IDE "preferences.txt" file which is shared with any other IDE version installations.

A better way to install the IDE, especially if you want to expand its scope and add your own libraries and hardware definitions, is to use the "Portable" method. A "Portable" IDE is, to all intents and purposes, a stand-alone IDE that uses little of the user's files and folders within Windows. A major problem with the older pre 1.6.0 IDE was that temporary files were left in the user's "Temp" folder and they gradually multiplied. The newer IDE now cleans up after itself and leaves little behind.

Another benefit of the Portable IDE is that the user's "preferences.txt" file is stored within the IDE's environment and is specific to that installation of the IDE. It is possible to have multiple installations (even versions) of the IDE each with their own Preferences file. For example, I also use the Arduino IDE to program ESP8266 Wi-Fi modules but prefer not to add their definitions to my AVR IDE. Using the Portable approach I am able to have two separate IDE installations, one for AVRs and one for the ESP8266. They have their own Preferences, sketchbooks and library locations.

Here's how do a Portable installation:

  1. Download the Windows ZIP file instead of the Windows Installer to your Downloads folder
  2. Unzip or extract the zipped files to a location of your choice. I recommend the root of your C:\ drive. You should end up with a folder on your C:\ drive called "C:\arduino-<version number>"
  3. Open the "C:\arduino-<version number>" folder and create a folder called "portable" (without the quotes)

Please Note: You may have to manually install the generic Arduino FTDI drivers which are located within the IDE folder.

Create a shortcut on your Desktop to the file called "arduino.exe" in your new Arduino folder and, as soon as you start the application, the IDE will know to use the "portable" folder to store its working files in.

Magic will happen and if you now open the "portable" folder it will now contain the following:


If you update your version of the IDE to a new release you only have to copy the "portable" folder to your new IDE folder and all your settings and additions will be available to your new IDE. Also note that the preferences.txt file is within the IDE's environment and will not be used if you install a second or subsequent IDE.

The "sketchbook" folder is where your sketches will be saved and the "preferences.txt" file contains all the settings the you configure in the IDE's Preferences setup. If you open the "sketchbook" folder you will find that it contains a folder named "libraries" which is where you should place any libraries that you download.

It doesn't stop there! If you use the "Boards Manager" tool, any additional board types that you download will be placed into a new folder called "packages" within the "portable" folder. If you use the "Library Manager" tool new or updated libraries will be placed in a folder named "staging" within the "portable" folder.

What you end up with is a fully (well almost) self-contained IDE which can even be put onto a USB Flash Drive if you so wish.

From now on, I shall only refer to the Portable IDE so, here is what the unzipped arduino folder should now contain:

C:\  // The root of your hard drive
Folder Icon arduino-1.x.x  // The "Arduino environment"
Folder Icon drivers  // FTDI USB Driver files for the UNO etc.
Folder Icon examples  // The example sketches such as "Blink" etc.
Folder Icon hardware  // Definitions for board types, core files and programmers etc.  
Folder Icon java  // IDE support files
Folder Icon lib  // IDE support files
Folder Icon libraries  // The supplied libraries
    Folder Icon portable  // The new folder described above
Folder Icon reference  // Arduino "Reference" files
Folder Icon tools  // Support files
File Icon arduino.exe  // The Arduino IDE application
    File Icon arduino_debug.exe  // Runs the IDE with a "debug" window for IDE error reporting
    File Icon cyggcc_s-1.dll  // Support file
    File Icon cygiconv-2.dll  // Support file
    File Icon cygwin1.dll  // Support file
    File Icon cygz.dll  // Support file
    File Icon libusb0.dll  // Support file
    File Icon msvcp100.dll  // Support file
    File Icon msvcr100.dll  // Support file
File Icon revisions.txt  // Application specific changes etc.

And Finally, Preferences...

Preferences WindowOpen the Arduino IDE and select "File/Preferences" and the window shown to the left will appear.

The selections shown are the default ones that I use. Please note that I do not select "Check for updates on startup" and "Save when verifying or upload", here's why.

"Check for updates on startup" tells the IDE to check online for new board definitions and libraries. This may seem to be a good idea and it can be but, it can also be very awkward if, like me, you edit your boards entries or have libraries under development because the IDE will attempt to overwrite your selections or nag you that a library is not compliant etc.

I leave you to decide, try it and see what happens!

"Save when verifying or upload" may seem a good idea but it can become annoying and destructive if you do a test verify or test out a code change before saving your changes. With this option selected it's too late as the sketch is automatically saved to disc every time you verify or upload.

Trust me, it gets very annoying!