AVR SD Hex Loader ISP

This project is all about getting your compiled sketch onto your target board without the need to have a connection to a computer running the Arduino IDE.

The video demonstrates the operation of the loader. The first board being loaded in the video is my DCSensor (Dust Collector Sensor) board. The DCSensor binary is loaded via its ICSP connector is 3202 bytes. The second board being loaded is my DCController board. The DCController binary is loaded via its TTL Serial connector is 61258 bytes.

Terms:

ICSP - In-Circuit Serial Programming. Many Arduino boards have a dedicated 6 pin 2x3 pin header labeled ICSP.

ISP - In-System Programmer.

Intel HEX - Intel hexadecimal object file format. A file format that conveys binary information in ASCII text form.

IDE - integrated Development Environment. In this specific case it’s the Arduino IDE.

AVR - not an acronym, it’s simply the name of a family of microcontrollers developed by Atmel and now owned by Microchip Technology that are found on many boards supported by the Arduino IDE.

stk500 - an ISP developed by Atmel. In this case it’s the set of documented commands supported by the ISP and many on-chip bootloaders.

SD - Secure Digital memory card format. This project uses a micro SD card.

Overview:

If you’ve ever looked into how the Arduino IDE works, you may know that the IDE uses the command line tool avrdude to convert the Intel HEX file containing your sketch’s binary into the stk500 commands needed to load your sketch onto the target AVR microcontroller. These commands are sent directly to the microcontroller if it has a bootloader, or through an ISP connected to the board’s ICSP connector. The ISP converts the stk500 commands into AVR serial programming instructions. Using either method, you need to have your board near the computer running the IDE. But what if your microcontroller is installed somewhere that it would be inconvenient to use your computer? (outside, dusty location, etc.) You would probably have to uninstall the board whenever you need to update the software. That’s where this project comes in. It allows you to load your microcontroller in-place without the need for a physical connection to the computer running the IDE. The SD Hex Loader ISP does what avrdude does only it doesn’t require any connection to a computer, just an SD card. The loader can also work as a standard USB AVR ISP.

You could also use this board for small scale production.

Update 4-Feb-2021: Added support for setting fuses and loading the bootloader associated with the sketch. For sketches that target a device with no bootloader, only the fuses and lock bits are set.

Update 6-Mar-2021: Updated URLs and most pictures to show the latest board version v1.4.

If anyone in the US is having trouble locating the ATmega644PA used in this project, I have plenty of them. Send me a message.

The workflows described above:

If you’re interested in building this project, the bare boards and parts list are available on PCBWay.

The code for this project is on GitHub

I’m not going to go into any detail about how to assemble the board. I’ve published several instructables that contain the steps needed to assemble a board. If you actually attempt to build this project and have questions, just send me a message (rather than a comment.) This instructable will describe how critical parts of the project work, how to operate it, and how to assemble the case.

The loader has four connectors for programming: two standard 6 pin TTL serial, and two standard 6 pin ISP, one of each for 3v3 and 5V. The ISP connects to the ICSP connector on the target board.

These connectors are buffered and switched using quad buffers. The buffers are used to isolate and switch the 5V signals and for level shifting 3v3 signals.

The AVR microcontroller is an ATmega644PA running at 5V,16MHz. This microcontroller has two serial USARTs, one SPI interface, and one I2C interface.

One of the serial USARTs is connected to a USB to TTL serial chip, the other is used for TTL serial programming.

The display module, micro SD connector, and the ISP connections are on the SPI bus.

As you may know, an ISP to ICSP connection is essentially SPI with the reset line serving as the chip select. The major twist is that the reset line must be held low for the duration of the programming session. As noted, the display board and SD card are also SPI. One of the quad buffers is used to disconnect the SPI bus from the ISP connectors when the display or SD card are accessed. In order for this to work, the MOSI and SCK lines on the ISP connectors have pull down resistors to keep these lines from floating when the output of the buffer isn’t enabled.

The I2C bus has a Real Time Clock chip and an optional connection for a serial EEPROM chip of the AT24C family. The reason for an RTC chip is explained later on. The EEPROM is for future development.

The board has Flat Cable Connectors for the SPI display and 5-button boards.

Other Boards:

- 240x240 RGB display board. Available on PCBWay

- 5 button board. Available on PCBWay

Status LEDs: (you only implement one)

If you plan to use the case I designed for the board, install the 5 pin PH2.0 connector on the board. You should not include the 4 status LEDs on the board.

If you plan to use the board without a case, you should include the 4 status LEDs on the board . You should not install the 5 pin PH2.0 connector.

This is an overview of the software used.

The only software that I didn’t write, other than the classes that are in the Arduino IDE bundle, are the classes that deal with the SD card.

- Bill Greiman's SdFat library. Copyright Bill Greiman 2011-2018 (currently using version 1.1.0)

My MacOS utility applications used in this project:

- HexLoaderUtility - used to copy Arduino generated hex files and extract configuration data.

- SubsetFontCreator - creates a subset of a font for either 1 bit or RGB displays as compressed bitmap data (RGB is antialiased, 1 bit isn’t).

- SerialHexLoader - used to debug and test the ISP. The app name is a bit confusing here. None of the SerialHexLoader HexLoader features are used, just the stk500 commands from the Command menu.

I’m describing the HexLoaderUtility application before describing the software on the board itself. Understanding what the utility application provides is needed in order understand how the software on the board uses it.

The SD card used by the SD Hex Loader ISP board contains Intel HEX and configuration files. The hex files are generated by the Arduino IDE in a temporary folder whenever you Verify or Upload a sketch. The IDE also creates many other files that contain information needed by the loader to target a board. To access these files I wrote a Mac OS application named HexLoaderUtility. If you looked at the first Instructable I published, AVR Programmer W/High Voltage, you would see that I dove in quite deeply into the files that the IDE creates. HexLoaderUtility leverages many of these sources to extract the information needed. In addition, HexLoaderUtility also parses the corresponding avrdude.conf file to get some of the configuration information. All of the accumulated configuration information is written out to a simple key/value text file read by the loader when working standalone in SD source mode.

For the HexLoaderUtility sources, built application, and instructions on how to use the utility application, go to the repository on GitHub.

The two major classes used on the board to load a hex file are AVRStreamISP and SDHexSession.

Class AVRStreamISP:

AVRStreamISP is heavily based on the original ArduinoISP sketch. Anyone comparing the sources can see the similarities. Apart from being a structured class, AVRStreamISP is a class that expects its data source to be a subclass of Stream. This was done so that the source of the data stream could be from an SD card or, as in the original ArduinoISP sketch, the Serial port. When a session starts, AVRStreamISP is given time via its Update routine. If there is anything in the stream to process, Update routes this to the appropriate function.

Class SDHexSession:

SDHexSession also requires a stream to operate. Whereas AVRStreamISP is primarily a consumer of stk500 commands, SDHexSession, like avrdude, is a producer of stk500 commands. SDHexSession opens the selected sketch’s hex file and its config file. Based on this it targets either the 2nd serial port (one of the TTL serial connectors), or the AVRStreamISP class (one of the ISP connectors.) The SDHexSession produces nearly the identical stream of stk500 commands as avrdude when loading a hex file to a target a microcontroller. It also mimics the timing needed to allow for the target to complete each command. After the hex file is loaded, SDHexSession verifies the load.

For the the complete set of project sources, go to the repository on GitHub .

Launch both the Arduino IDE and the HexLoaderUtility. Note that when the Arduino IDE quits it removes all of the temporary files created since the IDE launched. For this reason you need to keep the Arduino IDE running while using the HexLoaderUtility. The first time you run the HexLoaderUtility you’ll need to select the Arduino IDE and the folder where you would like to export the hex and configuration files.

In the Arduino IDE, select Verify for each of the sketches you’d like to get the hex files for. When you switch to the HexLoaderUtility, the list of sketches successfully verified will appear. Select the sketches to be exported, then right-click to display the contextual menu. Select Export from this menu. The hex and generated configuration files will appear in the designated Export folder.

For a slightly more detailed set of HexLoaderUtility instructions, see the readme in the GitHub repository.

Copy the pairs of hex/txt files to your SD card. Insert the card into the loader. Note that if the loader doesn't see a corresponding valid config file your sketch won't appear in the loader UI.

When you turn on the loader the main screen prompts you choose the source, USB, SD, or SDBL. Selecting SD or SDBL will prompt you to insert an SD card whenever the card isn’t inserted.

In USB source mode the loader behaves the same as the original ArduinoISP. All stk500 commands received via USB serial are blindly interpreted and passed through to either of the ISP connectors. The ISP outputs are mirrored, no UI selection for 3v3 or 5V is needed.

In SD source mode you’re expected to select the filename of the sketch to be loaded onto the target board. Use the Down button to move the selection frame to the sketch filename. Use either the Left or Right arrows to cycle through the names of sketches on the SD card. The selected sketch is the sketch currently displayed, no other action is needed to formally select a sketch. By default the loader determines based on the associated configuration whether ISP or TTL Serial connectors are used. ISP is used when the upload speed value in the corresponding config file is zero (as it would be for something without a bootloader.) You can also force ISP to be used via a setting in the settings screen. You would do this if the target board only has a USB serial connection but also has an ICSP connector, as most do.

In SDBL source mode you’re expected to select the filename of the sketch for which the fuses and bootloader should be loaded. The target for this mode is always the ISP. This operation will erase the entire chip, including resetting the lock bits. The fuses and bootloader to be loaded are determined by the HexLoaderUtility and saved to the sketch's configuration file when the sketch is exported. HexLoaderUtility maintains the export/bootloaders folder containing all of the bootloaders for the exported sketches.

Once you’ve selected the sketch, press the up button to move to select Start Serial or Start ISP.

If the connection is OK, the Writing and Verifying percentages followed by a ”Success!” message.

That’s basically it, pretty simple.

Settings:
Pressing the up button from the first line of the main screen will display the settings. In here you can set the RTC time, whether to force the use of the ISP connection, and the clock speed used. The clock speed is only used when the source is USB. The default is 1 MHz, which is what you would use if you’re setting fuses and/or loading a bootloader for the first time from the Arduino IDE or directly via avrdude. If you’re loading a sketch you should set this to the clock frequency of the target microcontroller. Doing this will greatly speed up loading and verification. This is one of the advantages over the original ArduinoISP. Because the original ArduinoISP has no user interface, the author had to assume a 1 MHz clock. To get back to the main screen press the up arrow till the main screen appears.

Why the loader has a real time clock:
Aside from using the RTC for determining when to put the display to sleep, which could simply be done by checking millis(), the RTC is used by the SDHexSession to replace a specific memory location on the target with the Unix time the load occurred. This feature only effects a specific sketch. It was implemented because one of my boards needs to generate a unique ID. This feature isn’t exposed in the HexLoaderUtility which determines the memory address to be replaced or by anything in the loader user interface. This proof of concept feature could be expanded to replace flash memory with any value desired. For example, to automatically write a unique serial number for each load. Send me a message if you have any questions or you would like more information.

The case was designed using Fusion 360 using the Eagle feature where you can upload your board(s). Many of the parts I use don’t have a 3D representation so I created them in Fusion 360 and imported them into my library on Eagle. I found it’s very much worth the effort.

Here’s a few of the 3D parts I created in Fusion 360 and boards that I’ve uploaded from Eagle:

Taking the time to do this allows you to place your boards in the case design without having to create a component representing its geometry and the location of all of the parts on the board. If you update the location of a part in Eagle you just sync the changes with Fusion.

Print the 3D case parts. (attached below).

I don't have the greatest 3D printer. I printed the case using PLA at 80% fill, draft quality, support everywhere.

Clean out the supports and check to see if the buttons on the 5 button board function freely. If not check to make sure the buttons are accurately centered on the board. You may also need to ream out the holes depending on the accuracy of your printer.

Ream the 3mm status LED holes using a 3mm drill bit.

Do NOT ream the 2mm screw holes for the boards and case bottom.

The display board is connected to the loader board using an 8 pin 10cm 1mm pitch FFC cable A/B.

The 5 button board is connected to the loader board using a 6 pin 10cm 1mm pitch FFC cable A/B.

Create a 6.5cm long 4 LED status assembly using the ground wire common to all 3mm LEDs wired as shown in the schematic attached below.

Green - Power on
Blue - Heartbeat
Yellow - In Programming Mode
Red - Error

Note that the current limiting resistors are on the loader board.

Carefully place plastic epoxy around the base of each LED, then insert the LEDs as shown. Set aside to cure.

Attach the 6 pin 10cm 1mm pitch FFC cable to the 5 button board.

Install the board in the case using 4 M2x4 button head screws.

Attach the 8 pin 10cm 1mm pitch FFC cable to the display board.

Install the board in the case using 4 M2x4 button head screws.

Connect the LED status cable to the loader board.

Connect the two FFC cables to the loader board.

Mount the loader board to the case back using 4 M2x4 button head screws.

Mount the case back to the case top using 4 M2x5 flat head screws.

Note: The small hole near the display is for resetting the board. I made a loft with a hole in it that ends just above the reset button. Sticking an ordinary pin in the hole will reset the board without any chance of shorting anything on the board.

Things fixed in board version 1.4: (original published version is 1.3)

- The LED connector has been mirrored: The status LED PH2.0 connector is now mounted on the underside/back of the board.

- The RTC battery holder has been moved so that it doesn't interfere with the display connector. In version 1.3 I had to break off the corner of the CR2032 battery holder closest to the 8 pin display connector in order to open the display connector. In version 1.3 the battery holder was also shifted down and to the right on its pads, as far away from the display connector as possible. v1.3 is shown below.

- The pin connected to the heartbeat status LED is now on a timer that allows it to fade in and out as expected. In v1.3 the heartbeat LED was on a timer used by the RTC so it could only blink. In v1.4 it's on PD4.

I hope this project gives you some insight into how the Arduino IDE works. If you have specific questions send me a message.