iot for dummies | part 3

[or] putting files into spiffs

2019/07/29

Tags: code diy electronics

All lessons (and the project files):

  1. First steps
  2. Coding over-the-air
  3. Putting files into SPIFFS
  4. Untethered serial monitor

You remember last time I told you we’re done with tethering our Arduino with a USB cable. Well, I kind of cheated a bit. You see, indeed if you only need to upload instructions to Arduino over-the-air, you can always do that with the ArduinoOTA, as we learned last time, and you don’t ever need to connect to your microcontroller through a USB adapter. However, Arduinos also have a built-in (SPI) flash memory (a very limited in capacity) which can be used to store permanent information. Like a website, or a database, or an icon, or some text! This memory is non-volatile, meaning that whatever its content is, when Arduino loses its power the data does not evaporate.

If we want our Arduino to work as a fully functional server we need to teach it how to upload and write files to the non-volatile memory, namely the html pages, js scripts, databases, etc.

And again, since we’re lazy and don’t want to tether Arduino too much, we’ll just do it once to upload “the uploader” itself, an html website that will serve as a file uploader in the future! And then we’ll teach Arduino how to treat the uploaded files and write them to SPIFFS. For more details you can actually read this guy who seems to understand a lot more in the matter than I do. And here I’ll just try to adapt his solution to a more user-friendly manner and finally provide you a promised over-the-air debugger!

Now this might sound a bit tricky and difficult, and it actually is… But bare with me, as the results totally worth it! Let’s move on.

result-pt3

So the result for this lesson would be to set up a server that will handle file uploading to the SPIFFS memory.

Upload the uploader.html

First create an uploader.html document with the following content:

<form method="post" enctype="multipart/form-data">
  <input type="file" name="name">
  <input class="button" type="submit" value="Upload">
</form>

This simply contains a file uploader field and a submit button that we’ll be treating later in the Arduino code. We’ll use this page to upload all the other files to our permanent SPIFFS memory.

Now we need to somehow put this file into our Arduino SPI flash memory. And this is the only time you might want to use [Arduino IDE](https://www.arduino.cc/en/main/software) (which I honestly hate for its inconvenience and shitty design). The instructions on how to do that can be found here, but overall here’s what you need to do:

  1. create a new sketch in Arduino IDE;
  2. make a new data folder in the sketch folder and place our uploader.html inside;
  3. connect our NodeMCU Arduino via USB;
  4. in Arduino IDE go to Tools > Board, select the proper board (if you don’t see the one you need, here’s a tutorial on how to fix that);
  5. Tools > Port, and select the port to which your Arduino is connected to;
  6. in the Tools > Flash size select 4M (1M SPIFFS);
  7. now upload an empty sketch to Arduino;
  8. then Tools > ESP8266 Sketch Data Upload.

If you don’t have the appropriate board, you can easily install it by going to Tools > Board > Board Manager... and selecting the esp8266 one. If you still struggle to Verify the code and get the Error compiling ... message, please check this link for resolution.

This will upload the content of data folder (in this case our uploader.html file) to Arduino’s SPIFFS memory. Now you can finally disconnect the USB and through it away (just kidding you might need it to charge your ps4 controllers, so don’t just throw it). Since this might be a bit tricky, the whole process is demonstrated in the animation below.

Setting up an uploader

Once the uploader.html is in the permanent memory, and our Arduino has a connection with WiFi and an OTA running (also make sure it has a power supply connected) we can teach it how to treat the new uploaded files (like html-s or css-s or text-s, anything) and write them properly into the memory.

I’ll be using the code we wrote on the previous lesson and will just add the instructions for our humble “server” to treat the file uploading and writing. So here’s what the whole code will look like if you put everything together:

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ArduinoOTA.h>

#include <ESP8266HTTPClient.h>
#include <ESP8266WebServer.h>
#include <FS.h>

/* declaring the server */
ESP8266WebServer server(80);
/* global variable to store the uploaded file */
File fsUploadFile;
/* auxiliary functions to read and interpret uploaded files */
bool handleFileRead(String path);
void handleFileUpload();
String getContentType(String filename);

const char * SSID = "<YOUR_WIFI_SSID>";
const char * PASS = "<YOUR_WIFI_PASS>";

#include "Initialize.h"

void setup() {
  Serial.begin(74880);
  initializeWifi();
  initializeOTA();

  SPIFFS.begin(); // initialize SPIFFS
  /* initialize server
  ... here we teach Arduino how to be a server...
  ... i.e. how to handle clients' requests */
  server.on("/upload", HTTP_GET, []() {
    if (!handleFileRead("/uploader.html"))
      server.send(404, "text/plain", "404: Not Found");
  });
  server.on("/upload", HTTP_POST,
    [](){ server.send(200); },
    handleFileUpload
  );
  server.onNotFound([]() {
    if (!handleFileRead(server.uri()))
      server.send(404, "text/plain", "404: Not Found");
  });
  server.begin();
}

void loop() {
  ArduinoOTA.handle();
  server.handleClient();
}

bool handleFileRead(String path) { // send the right file to the client (if it exists)
  Serial.println("handleFileRead: " + path);
  if (path.endsWith("/")) path += "index.html";          // If a folder is requested, send the index file
  String contentType = getContentType(path);             // Get the MIME type
  String pathWithGz = path + ".gz";
  if (SPIFFS.exists(pathWithGz) || SPIFFS.exists(path)) { // If the file exists, either as a compressed archive, or normal
    if (SPIFFS.exists(pathWithGz))                         // If there's a compressed version available
      path += ".gz";                                         // Use the compressed verion
    File file = SPIFFS.open(path, "r");                    // Open the file
    size_t sent = server.streamFile(file, contentType);    // Send it to the client
    file.close();                                          // Close the file again
    Serial.println(String("\tSent file: ") + path);
    return true;
  }
  Serial.println(String("\tFile Not Found: ") + path);   // If the file doesn't exist, return false
  return false;
}

void handleFileUpload() { // upload a new file to the SPIFFS
  HTTPUpload& upload = server.upload();
  if(upload.status == UPLOAD_FILE_START){
    String filename = upload.filename;
    if(!filename.startsWith("/")) filename = "/"+filename;
    Serial.print("handleFileUpload Name: "); Serial.println(filename);
    fsUploadFile = SPIFFS.open(filename, "w");            // Open the file for writing in SPIFFS (create if it doesn't exist)
    filename = String();
  } else if(upload.status == UPLOAD_FILE_WRITE){
    if(fsUploadFile)
      fsUploadFile.write(upload.buf, upload.currentSize); // Write the received bytes to the file
  } else if(upload.status == UPLOAD_FILE_END){
    if(fsUploadFile) {                                    // If the file was successfully created
      fsUploadFile.close();                               // Close the file again
      Serial.print("handleFileUpload Size: "); Serial.println(upload.totalSize);
      server.sendHeader("Location","/success.html");      // Redirect the client to the success page
      server.send(303);
    } else {
      server.send(500, "text/plain", "500: couldn't create file");
    }
  }
}

String getContentType(String filename) { // convert the file extension to the MIME type
  if (filename.endsWith(".html")) return "text/html";
  else if (filename.endsWith(".css")) return "text/css";
  else if (filename.endsWith(".js")) return "application/javascript";
  else if (filename.endsWith(".ico")) return "image/x-icon";
  else if (filename.endsWith(".gz")) return "application/x-gzip";
  return "text/plain";
}

Upload this code over the air and take a closer look at it once it’s done. Feel free to read the comments in the code, they’re pretty self-descriptive. But overall, what we’re doing here, is teaching the Arduino how to be a server by telling it what to do with the clients’ requests (these server.on-s are exactly that). Since Arduino also has no clue how to read the files from its memory and how to display them on requests – we really need to teach it.

Once the upload is done you can enter the following link in your browser 192.168.0.1/upload (paste whatever local IP address your Arduino had), and you should see the following page:

Of course, since our humble server uses an http protocol, the browser complains that the connection is not secure.

You can now upload any files to your SPIFFS memory just by accessing this page, but be careful, as the memory capacity is pretty limited, and if overflown, you’d need to free it up by connecting to USB and uploading an empty data folder.

Alternatively you can actually put something like this: SPIFFS.remove("/<bad_file>.html"); somewhere in the setup() next time you upload a new code to Arduino, and once Arduino loads it will try to delete this <bad_file>.html from its memory.

Making things neat

Now you might want to put all these gory functions to some header file to hide them away, and this time I’ll take the OOP kind of approach, which might turn out to be useful in the future.

I put all the server-client related stuff into the include/Myerver.h header file, which we’ll be populating with much more exciting commands in the future:

#ifndef MYSERVER_H
#define MYSERVER_H

File fsUploadFile;
String getContentType(String filename);
bool handleFileRead(String path);
void handleFileUpload();

struct ClientServer {
  ClientServer();
  void initialize();
  void handle();
} myClientServer;

ClientServer::ClientServer() {}

void ClientServer::initialize() {
  server.on("/upload", HTTP_GET, []() {
    if (!handleFileRead("/uploader.html"))
      server.send(404, "text/plain", "404: Not Found");
  });
  server.on("/upload", HTTP_POST,
    [](){ server.send(200); },
    handleFileUpload
  );
  server.onNotFound([]() {
    if (!handleFileRead(server.uri()))
      server.send(404, "text/plain", "404: Not Found");
  });
  server.begin();
}

void ClientServer::handle() {
  server.handleClient();
}

String getContentType(String filename) { // convert the file extension to the MIME type
  if (filename.endsWith(".html")) return "text/html";
  else if (filename.endsWith(".css")) return "text/css";
  else if (filename.endsWith(".js")) return "application/javascript";
  else if (filename.endsWith(".ico")) return "image/x-icon";
  else if (filename.endsWith(".gz")) return "application/x-gzip";
  return "text/plain";
}

bool handleFileRead(String path) { // send the right file to the client (if it exists)
  Serial.println("handleFileRead: " + path);
  if (path.endsWith("/")) path += "index.html";          // If a folder is requested, send the index file
  String contentType = getContentType(path);             // Get the MIME type
  String pathWithGz = path + ".gz";
  if (SPIFFS.exists(pathWithGz) || SPIFFS.exists(path)) { // If the file exists, either as a compressed archive, or normal
    if (SPIFFS.exists(pathWithGz))                         // If there's a compressed version available
      path += ".gz";                                         // Use the compressed verion
    File file = SPIFFS.open(path, "r");                    // Open the file
    size_t sent = server.streamFile(file, contentType);    // Send it to the client
    file.close();                                          // Close the file again
    Serial.println(String("\tSent file: ") + path);
    return true;
  }
  Serial.println(String("\tFile Not Found: ") + path);   // If the file doesn't exist, return false
  return false;
}

void handleFileUpload(){ // upload a new file to the SPIFFS
  HTTPUpload& upload = server.upload();
  if(upload.status == UPLOAD_FILE_START){
    String filename = upload.filename;
    if(!filename.startsWith("/")) filename = "/"+filename;
    Serial.print("handleFileUpload Name: "); Serial.println(filename);
    fsUploadFile = SPIFFS.open(filename, "w");            // Open the file for writing in SPIFFS (create if it doesn't exist)
    filename = String();
  } else if(upload.status == UPLOAD_FILE_WRITE){
    if(fsUploadFile)
      fsUploadFile.write(upload.buf, upload.currentSize); // Write the received bytes to the file
  } else if(upload.status == UPLOAD_FILE_END){
    if(fsUploadFile) {                                    // If the file was successfully created
      fsUploadFile.close();                               // Close the file again
      Serial.print("handleFileUpload Size: "); Serial.println(upload.totalSize);
      server.sendHeader("Location","/success.html");      // Redirect the client to the success page
      server.send(303);
    } else {
      server.send(500, "text/plain", "500: couldn't create file");
    }
  }
}

#endif

And now our main.cpp looks way cleaner (of course, don’t forget the Initialize.h from the previous lesson):

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ArduinoOTA.h>

#include <ESP8266HTTPClient.h>
#include <ESP8266WebServer.h>
#include <FS.h>

ESP8266WebServer server(80);
const char * SSID = "<YOUR_WIFI_SSID>";
const char * PASS = "<YOUR_WIFI_PASS>";

#include "Initialize.h"
#include "Server.h"

void setup() {
  Serial.begin(74880);
  initializeWifi();
  initializeOTA();

  SPIFFS.begin();
  myClientServer.initialize();
}

void loop() {
  ArduinoOTA.handle();
  myClientServer.handle();
}

There is also a useful function which you can use to access the current content of the SPIFFS filesystem. You can add it as a method for your ClientServer, and then Serial.println(myClientServer.getFilesystem()) when needed:

String ClientServer::getFilesystem() {
  Dir dir = SPIFFS.openDir("/");
  String str = "";
  while (dir.next()) {
      str += dir.fileName();
      str += " / ";
      str += dir.fileSize();
      str += "\r\n";
  }
  return str;
}
cd ~